Google App Engine: Import CSV to Datastore

Processing CSV Data with the Google App Engine is a two step process.
First you need to upload the data and store it in the blobstore.
Then you retrieve the file and process the information and save it to the Datastore(Database).
You can process the csv data with the standard python tools from the csv module, however instead of parsing a file you must parse a BlobReader.

Posted by happyneal

  • Angel

    Just wanted to say GOD BLESS YOU!!! for this post. I’ve been banging my head against this for three days. GAE really lacks tools for easy import/export of datastore data. I’m not even sure they still support the BulkLoader anymore. I had a heck of a time trying to find any useful info on how to do this then I stumbled upon your awesome post. Thanks again…really!!!

  • This has helped me so much! BulkLoader was not working for me AT ALL and after hours of searching when trying to deploy my first app, this post was my saving grace. Thanks for taking the time to share!

  • Annokin

    You’re a life saver man!
    Thank you so much! 😀

  • MATTM

    Thank you for this. Im new to GAE and fairly new to Python, but with your example I’m almost accomplishing what I set out to do.

    I am however at a standstill. Maybe you can help…

    I keep getting an error…’ValueError: need more than 2 values to unpack’ at the line…’line 49, in process_csv….date, data, value = row’

    Ive been trying to troubleshoot but am stuck.

    My code is as follows…any assistance is greatly appreciated.

    #!/usr/bin/env python
    # encoding: utf-8

    import webapp2
    import csv
    import wsgiref.handlers
    import time
    from google.appengine.ext import blobstore
    from google.appengine.ext.webapp import blobstore_handlers
    from google.appengine.ext import db

    class MyHandler(webapp2.RequestHandler):
    def get(self):
    upload_url = blobstore.create_upload_url(‘/upload’)

    html_string = “””

    Upload File:

    “”” % upload_url

    self.response.out.write(html_string)

    class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
    def post(self):
    upload_files = self.get_uploads(‘file’)
    blob_info = upload_files[0]
    process_csv(blob_info)
    blobstore.delete(blob_info.key())
    self.redirect(“/”)

    def process_csv(blob_info):
    blob_reader = blobstore.BlobReader(blob_info.key())
    reader = csv.reader(blob_reader, delimiter=’,’, quotechar=’|’)

    #issue has to do with the next line
    for row in reader:
    date, data, value = row
    entry = EntriesDB(date=date, data=data, value=int(value))
    entry.put()

    class EntriesDB(db.Model):
    date = db.DateProperty()
    data = db.StringProperty()
    value = db.IntegerProperty()

    def main():
    app = webapp2.WSGIApplication([
    (‘/’, MyHandler),
    (‘/upload’, UploadHandler)
    ], debug=True)

    wsgiref.handlers.CGIHandler().run(app)

    if __name__ == ‘__main__’:
    main()

    • date, data, value = row

      is causing the error, if one of the values is not set the variables cannot be unpacked. So something is wrong with your data. You should add some error handling to handle faulty data.

  • zih

    Hello! Your example here has been really useful in helping me with a project I’m working on! May I ask how I can print the content of the uploaded cvs file?

    I tried following the example here
    https://webapp-improved.appspot.com/tutorials/gettingstarted/usingdatastore.html

    I simplified the EntriesDB to just having data. And made sure the CSV file for testing only has a column of strings. My code now simply adds the following (but only prints out ‘wrote’ multiple times):

    greetings = db.GqlQuery(“SELECT * ”
    “FROM EntriesDB “)
    for g in greetings:
    self.response.out.write(‘%s wrote:’ % g.data())

    Is there perhaps some resource you can point me to get started on printing the user uploaded CSV file?
    Thanks a lot in advance!