Overview
The module/script pysql
can be used to import data into an Oracle
database. It reads pysql
files which are a variant of normal Oracle SQL
files.
A PySQL file can contain two different types of commands.
SQL commands
A PySQL file may contains normal SQL commands. For the pysql
script
to be able to execute these commands they must be terminated with a comment
line that starts with -- @@@
. pysql
will strip off a trailing
;
or /
from the command and execute it. Any exception that is raised
as a result of executing the command will stop the script and be reported.
This is in contrast to how sqlplus
executes SQL commands. sqlplus
would continue after an error and exit with status code 0 even if there were
errors. It is also possible to explicitely ignore any exception raised by the
command by specifying a different exception handling mode.
A PySQL file that only contains SQL commands is still a valid SQL file from
the perspective of Oracle, so it still can be executed via sqlplus
.
PySQL commands
A PySQL file may also contain PySQL commands. A PySQL command looks like a
Python dictionary literal. This literal must either be contained in a single
line or it must start with a line that only contains {
and end at a
line that only contains }
.
For further information about the different commands and which keys they support,
see the class Command
and its subclasses.
Example
The following is a complete PySQL file that will create a sequence, table and procedure and will call the procedure to insert data into the table:
create sequence person_seq increment by 10 start with 10 maxvalue 1.0e28 minvalue 10 nocycle cache 20 noorder ; -- @@@ create sequence contact_seq increment by 10 start with 10 maxvalue 1.0e28 minvalue 10 nocycle cache 20 noorder ; -- @@@ person: table create table person ( per_id integer not null, per_firstnane varchar2(200), per_lastnane varchar2(200) ); -- @@@ person: primary key alter table person add constraint person_pk primary key(per_id); -- @@@ contact: table create table contact ( con_id integer not null, per_id integer not null, con_type varchar2(200), con_value varchar2(200) ); -- @@@ contact: primary key alter table contact add constraint contact_pk primary key(con_id); -- @@@ person: insert procedure create or replace procedure person_insert ( c_user in varchar2, p_per_id in out integer, p_per_firstnane in varchar2 := null, p_per_lastnane in varchar2 := null ) as begin if p_per_id is null then select person_seq.nextval into p_per_id from dual; end if; insert into person ( per_id, per_firstnane, per_lastnane ) values ( p_per_id, p_per_firstnane, p_per_lastnane ); end; / -- @@@ contact: insert procedure create or replace procedure contact_insert ( c_user in varchar2, p_con_id in out integer, p_per_id in integer := null, p_con_type in varchar2 := null, p_con_value in varchar2 := null ) as begin if p_con_id is null then select contact_seq.nextval into p_con_id from dual; end if; insert into contact ( con_id, per_id, con_type, con_value ) values ( p_con_id, p_per_id, p_con_type, p_con_value ); end; / -- @@@ import data { 'type': 'procedure', 'name': 'person_insert', 'args': { 'per_id': var('per_id_max'), 'per_firstname': 'Max', 'per_lastname': 'Mustermann', } } { 'type': 'procedure', 'name': 'contact_insert', 'args': { 'per_id': var('per_id_max'), 'con_id': var('con_id_max'), 'con_type': 'email', 'con_value': 'max@example.org', } } { 'type': 'file', 'name': 'portrait_{per_id_max}.png', 'content': b'\x89PNG\r\n\x1a\n...', } { 'type': 'resetsequence', 'sequence': 'person_seq', 'table': 'person', 'field': 'per_id', } {"type": "compileall"} {"type": "checkerrors"}
This file can then be imported into an Oracle database with the following command:
python pysql.py user/pwd@database data.pysql
This will create two sequences, two tables and two procedures. Then it will
import two records, one by calling person_insert
and one by calling
contact_insert
. The PL/SQL equivalent of the above is:
declare v_per_id_max integer; v_con_id_max integer; begin person_insert( per_id=v_per_id_max, per_firstname='Max', per_lastname='Mustermann' ); contact_insert( con_id=v_con_id_max, per_id=v_per_id_max, con_type='email', con_value='max@example.org' ) end;
Furthermore it will create one file (named something like portrait_42.png
)
and reset the sequence person_seq
to the maximum value of the field
per_id
in the table person
.
Variables
Variable objects can be used to receive out parameters of procedure calls or
SQL statements. A variable object can be specified like this var("foo")
.
"foo"
is the "name" of the variable. When a variable object is passed
to a procedure the first time (i.e. the variable object is uninitialized),
a cx_Oracle
var
object will be passed and the resulting value after
the call will be stored under the name of the variable. When the variable is
used in a later command the stored value will be used instead. (Note that it's
not possible to use the same variable twice in the same procedure call,
if it hasn't been used before, however in later commands this is no problem).
The type of the variable defaults to int
, but a different type can be passed
when creating the object like this: var("foo", str)
.
It is also possible to create variable objects via command line parameter.
As a PySQL command is a Python literal, it is possible to use Python expressions
inside a PySQL command. A variable object that has a value will be replaced by
that value in such an expression, so stuff like 2*var("foo")
can be used.
An uninitialized variable object is considered "false", This makes it possible to default to another value if a variable is uninitialized:
var('foo', int) or 42
External files
Inside a PySQL command it is possible to load values from external files.
The loadbytes
function loads a bytes
object from an external file
like this:
loadbytes("path/to/file.png")
A string can be loaded with the loadstr
function like this:
loadstr("path/to/file.txt", "utf-8", "replace")
The second and third argument are the encoding and error handling name respectively.
The filename is treated as being relative to the file containing the
loadbytes
or loadstr
call.
Command line usage
pysql.py
has no external dependencies except for cx_Oracle
and can
be used as a script for importing a PySQL file into the database. As a script
it supports the following command line options:
connectstring
An Oracle connectstring.
file
The name of the PySQL file that will be read and imported. If
file
isn't specified the commands are read fromstdin
.-v
,--verbose
Gives different levels of output while data is being imported to the database. Possible levels are:
0
(no output),1
(one dot for each command),2
(each command name) or3
(detailed output for each command/procedure call)-z
,--summary
Give a summary of the number of commands executed and procedures called.
-c
,--commit
Specifies when to commit database transactions.
record
commits after every command.once
(the default) at the end of the script andnever
rolls back the transaction after all commands.-s
,--scpdirectory
The base directory for
scp
file copy commands. As files are copied viascp
this can be a remote filename (likeroot@www.example.org:~/uploads/
) and must include a trailing/
.-f
,--filedirectory
The base directory for the
file
file save commands. It must include a trailing/
.-t
,--terminator
The terminator after an SQL command (should be a valid SQL comment; default
-- @@@
).-D
,--define
Can be used multiple times to define variables. Supported formats are:
name
Defines a string variable named
name
and sets the value to the empty string.name=value
Defines a string variable named
name
and sets the value tovalue
.name:type
Defines a variable named
name
of typetype
and sets the value toFalse
,0
,0.0
or the empty string depending on the type. Supported types arestr
,bool
,int
andfloat
.name:type=value
Defines a variable named
name
of typetype
and sets the value tovalue
. For typebool
supprted values are0
,no
,false
,False
,1
,yes
,true
andTrue
.
def format_class
(obj
):
class Context
(object
):
A Context
objects contains the configuration and run time information
required for importing a PySQL file.
def __init__
(self
, db
=None
, scpdirectory
=''
, filedirectory
=''
, commit
='once'
, terminator
='-- @@@'
, raiseexceptions
=True
, verbose
=0
, summary
=False
, vars
=None
):
def var
(self
, key
, type
=<class 'int'>
):
def loadbytes
(self
, filename
):
def loadstr
(self
, filename
, encoding
=None
, errors
='strict'
):
def _load
(self
, stream
):
Load a PySQL file from stream
. stream
must be an iterable
over lines that contain the PySQL commands.
This function is a generator. Its output are the PySQL command objects
(i.e. instances of Command
).
def executeall
(self
, stream
):
Execute all command in stream
. stream
must be an iterable
over lines that contain the PySQL commands.
def _printsummary
(self
):
class Command
(object
):
The base class of all commands. A Command
object is created from
a command dictionary literal in a PySQL file. The keys in the command
dictorionary that are supported by all command types are the following:
type
(string (optional))This is either
"procedure"
(the default),"sql"
,"file"
,"scp"
,"resetsequence"
,"setvar"
,"include"
,"compileall"
or"checkerrors"
and specifies the type of the PySQL command.raiseexceptions
(bool (optional))Specifies whether exceptions that happen during the execution of the command should be reported and terminate the script (
True
, the default), or should be ignored (False
).
def __init__
(self
, location
, raiseexceptions
):
def __str__
(self
):
def register
(cls
):
class IncludeCommand
(Command
):
The "include"
command includes another PySQL file. The filename is read
from the key "name"
. This name is interpreted as being relative to the
directory with the file containing the include
command.
def __init__
(self
, location
, raiseexceptions
, name
):
def __repr__
(self
):
def __str__
(self
):
class _SQLCommand
(Command
):
Common base class of ProcedureCommand
and SQLCommand
.
def _createvar
(cursor
, type
, value
):
def _executesql
(self
, context
, query
):
def _formatargs
(self
, context
):
class ProcedureCommand
(_SQLCommand
):
A "procedure"
command calls an Oracle procedure in the database.
In addition to "type"
and "raiseexceptions"
the following keys are
supported in the command dictionary:
name
(string (required))The name of the procedure to be called (This may include
.
for calling a procedure in a package or one owned by a different user).args
(dictionary (optional))A dictionary with the names of the parameters as keys and the parameter values as values. PySQL supports all types as values that
cx_Oracle
supports. In addition to those, three special classes are supported:sql
objects can be used to specify that the paramater should be literal SQL. So e.g.sql("sysdate")
will be the date when the PySQL script was executed.var
objects can be used to hold values that areOUT
parameters of the procedure. For example on first use ofvar("foo_10")
the value of theOUT
parameter will be stored under the key"foo_10"
. The next timevar("foo_10")
is encountered the value stored under the key"foo_10"
will be passed to the procedure. The type of the variable defaults toint
. If a different type is required it can be passed as the second argument tovar
, e.g.var("foo_10", str)
.Finally
loadbytes
andloadstr
objects can be used to load values from external files (as long as they are of typebytes
orstr
).loadbytes("foo/bar.txt")
will be replaced with the content of the external filefoo/bar.txt
(as abytes
object). If astr
object is required,loadstr
can be used. Encoding info can be passed like this:loadstr("foo/bar.txt", "utf-8", "replace")
def __init__
(self
, location
, raiseexceptions
, name
, args
=None
):
def __repr__
(self
):
def __str__
(self
):
def _formatprocedurecall
(self
, context
):
def execute
(self
, context
):
class SQLCommand
(_SQLCommand
):
An "sql"
command directly executes an SQL statement in the Oracle database.
In addition to "type"
and "raiseexceptions"
the following keys are
supported in the command dictionary:
sql
(string (required))The SQL to be executed. This may contain parameters in the form of
:paramname
. The values for those parameters will be taken fromargs
.args
(dictionary (optional))A dictionary with the names of the parameters as keys and the parameter values as values. Similar to procedure calls
var
andload
objects are supported. Howeversql
objects are not supported (they will be ignored).
def __init__
(self
, location
, raiseexceptions
, sql
, args
=None
):
def __repr__
(self
):
def __str__
(self
):
def _formatsql
(self
, context
):
def execute
(self
, context
):
class SetVarCommand
(Command
):
The "setvar"
command sets a variable to a fixed value. In addition to
"type"
and "raiseexceptions"
the following keys are supported in the
command dictionary:
name
: string (required)The name of the variable to set.
value
: (required)The value of the variable.
def __init__
(self
, location
, raiseexceptions
, name
, value
):
def __repr__
(self
):
def __str__
(self
):
def execute
(self
, context
):
class UnsetVarCommand
(Command
):
The "unsetvar"
command deletes a variable. In addition to "type"
and
"raiseexceptions"
the key name
is supported and must contain the
name of the variable.
def __init__
(self
, location
, raiseexceptions
, name
):
def __repr__
(self
):
def __str__
(self
):
def execute
(self
, context
):
class RaiseExceptionsCommand
(Command
):
The "raiseexceptions"
command changes the global error reporting mode
for all subsequent commands. After:
{"type": "raiseexceptions", "value": False}
for all subsequent commands any exception will be reported and command execution will continue with the next command.
{"type": "raiseexceptions", "value": True}
will switch back to aborting the execution of the PySQL script once an exception is encountered.
Note that the global configuration will only be relavant for commands that
don't specify the "raiseexceptions"
key themselves.
def __init__
(self
, location
, raiseexceptions
, value
):
def __repr__
(self
):
def __str__
(self
):
def execute
(self
, context
):
class CheckErrorsCommand
(Command
):
The "checkerrors"
command checks that there are no compilation errors in
the target schema. If there are, an exception will be raised. (The
raiseexceptions
key is supported, but its value will be ignored).
def __init__
(self
, location
, raiseexceptions
):
def __repr__
(self
):
def __str__
(self
):
def execute
(self
, context
):
class CompileAllCommand
(Command
):
The "compileall"
command will recompile all objects in the schema.
def __init__
(self
, location
, raiseexceptions
):
def __repr__
(self
):
def __str__
(self
):
def execute
(self
, context
):
class SCPCommand
(Command
):
The "scp"
command creates a file by copying it via the scp
command.
In addition to "type"
and "raiseexceptions"
the following keys are
supported in the command dictionary:
name
(string (required))The name of the file to be created. It may contain
format()
style specifications containing any key that appeared in a"procedure"
or"sql"
command. These specifiers will be replaced by the correct key values. As these files will be copied via thescp
command, ssh file names can be used.content
: bytes (required)The content of the file to be created. This can also be a
load
object, to load the content from an external file.
def __init__
(self
, location
, raiseexceptions
, name
, content
):
def __repr__
(self
):
def __str__
(self
):
def execute
(self
, context
):
class FileCommand
(Command
):
The "file"
command creates a file by directly saving it from Python.
In addition to "type"
and "raiseexceptions"
the following keys are
supported in the command dictionary:
name
(string (required))The name of the file to be created. It may contain
format()
style specifications containing any key that appeared in a"procedure"
or"sql"
command. These specifiers will be replaced by the correct key values.content
: bytes (required)The content of the file to be created. This can also be a
load
object, to load the content from an external file.mode
: integer (optional)The file mode for the new file. If the mode is specified
os.chmod
will be called on the file.owner
: integer or string (optional)The owner of the file (as a user name or a uid).
group
: integer or string (optional)The owning group of the file (as a group name or a gid). If
owner
orgroup
is given,os.chown
will be called on the file.
def __init__
(self
, location
, raiseexceptions
, name
, content
, mode
=None
, owner
=None
, group
=None
):
def __repr__
(self
):
def __str__
(self
):
def execute
(self
, context
):
class ResetSequenceCommand
(Command
):
The "resetsequence"
command resets a sequence in the Oracle database to
the maximum value of a field in a table. In addition to "type"
and
"raiseexceptions"
the following keys are supported in the command
dictionary:
sequence
: string (required)The name of the sequence to reset.
table
: string (required)The name of the table that contains the field.
field
: string (required)The name of the field in the table
table
. The sequence will be reset to a value, so that fetching the next value from the sequence will deliver a value that is larger than the maximum value of the fieldfield
in the tabletable
.minvalue
: integer (optional, default taken from sequence)The minimum value for the sequence.
increment
: integer (optional, default taken from sequence)The increment (i.e. the stop size) for the sequence.
def __init__
(self
, location
, raiseexceptions
, sequence
, table
, field
, minvalue
=None
, increment
=None
):
def __repr__
(self
):
def __str__
(self
):
def execute
(self
, context
):
class var
(object
):
var
instances are used to mark procedure values that are OUT
parameters. On first use the parameter is used as an OUT
parameter and
PySQL will remembers the OUT value under the unique key specified in the
constructor. When a var
object is used a second time its value will
be passed to the procedure as a normal IN
parameter.
Note that
def __init__
(self
, key
, type
=<class 'int'>
):
Create a var
instance. key
is a unique name for the value.
type
is the type of the value (defaulting to int
).
def __repr__
(self
):
def __bool__
(self
):
Variables without values are always false.
def _reprbytes
(value
):
def _reprstr
(value
):
class strvalue
(str
):
def __new__
(cls
, key
, value
=''
):
def __repr__
(self
):
class bytesvalue
(bytes
):
def __new__
(cls
, key
, value
=''
):
def __repr__
(self
):
class intvalue
(int
):
def __new__
(cls
, key
, value
=0
):
def __repr__
(self
):
class floatvalue
(float
):
def __new__
(cls
, key
, value
=0.0
):
def __repr__
(self
):
def _makevalue
(name
, value
):
class sql
(object
):
An sql
object can be used to specify an SQL expression as a
procedure parameter instead of a fixed value. For example passing the current
date (i.e. the date of the import) can be done with sql("sysdate")
.
def __init__
(self
, expression
):
def __repr__
(self
):
class loadbytes
(object
):
A loadbytes
object can be used to load a bytes
object
from an external file.
def __init__
(self
, filename
):
Create a new loadbytes
object. filename
is the name of the
file to be loaded. The filename is treated as being relative to the
directory containing the pysql file that contains the PySQL command with
the loadbytes
object.
def __repr__
(self
):
def execute
(self
, basefilename
):
Read the file and return the file content as a bytes
or
str
object. basefilename
is the filename containing the
PySQL command with the load
object (i.e. this determines the
base directory).
class loadstr
(object
):
A loadstr
object can be used to load a str
object
from an external file.
def __init__
(self
, filename
, encoding
=None
, errors
='strict'
):
Create a new loadbytes
object. filename
is the name of the
file to be loaded. The filename is treated as being relative to the
directory containing the pysql file that contains the PySQL command with
the loadstr
object. encoding
and errors
will be
used for the file content into a string.
def __repr__
(self
):
def execute
(self
, basefilename
):
Read the file and return the file content as a bytes
or
str
object. basefilename
is the filename containing the
PySQL command with the load
object (i.e. this determines the
base directory).
class loadedbytes
(bytes
):
def __new__
(cls
, filename
, value
):
def __repr__
(self
):
class loadedstr
(str
):
def __new__
(cls
, filename
, value
):
def __repr__
(self
):
class Error
(Exception
):
def __init__
(self
, location
):
def __str__
(self
):
class CompilationError
(Exception
):
Exception raised by CheckErrorsCommand
when invalid database
objects are encountered.
def __init__
(self
, objects
):
def __str__
(self
):
class SCPError
(Exception
):
Exception raised by SCPCommand
when a call to the scp
comamnd
fails.
def __init__
(self
, status
, msg
):
def __str__
(self
):
class Location
(object
):
The location of a PySQL/SQL command in a pysql file.