luigi.contrib.mysqldb module

class luigi.contrib.mysqldb.MySqlTarget(host, database, user, password, table, update_id, **cnx_kwargs)[source]

Bases: Target

Target for a resource in MySql.

Initializes a MySqlTarget instance.

Parameters:
marker_table = 'table_updates'
touch(connection=None)[source]

Mark this update as complete.

IMPORTANT, If the marker table doesn’t exist, the connection transaction will be aborted and the connection reset. Then the marker table will be created.

exists(connection=None)[source]

Returns True if the Target exists and False otherwise.

connect(autocommit=False)[source]
create_marker_table()[source]

Create marker table if it doesn’t exist.

Using a separate connection since the transaction might have to be reset.

class luigi.contrib.mysqldb.CopyToTable(*args, **kwargs)[source]

Bases: CopyToTable

Template task for inserting a data set into MySQL

Usage: Subclass and override the required host, database, user, password, table and columns attributes.

To customize how to access data from an input task, override the rows method with a generator that yields each row as a tuple with fields ordered according to columns.

rows()[source]

Return/yield tuples or lists corresponding to each row to be inserted.

output()[source]

Returns a MySqlTarget representing the inserted dataset.

Normally you don’t override this.

copy(cursor, file=None)[source]
run()[source]

Inserts data generated by rows() into target table.

If the target table doesn’t exist, self.create_table will be called to attempt to create the table.

Normally you don’t want to override this.

property bulk_size