'From Squeak3.1alpha of 6 February 2001 [latest update: #3611] on 17 February 2001 at 10:16:12 pm'!

!PyNode commentStamp: 'sma 2/17/2001 20:06' prior: 0!
Copyright (C) 2001 by Stefan Matthias Aust (sma@3plus4.de). All rights reserved.

I'm the abstract root for Python parse nodes.!


!PyAtom commentStamp: 'sma 2/17/2001 21:55' prior: 0!
Copyright (C) 2001 by Stefan Matthias Aust (sma@3plus4.de). All rights reserved.

I represent a Python literal.!


!PyObject commentStamp: 'sma 2/17/2001 22:05' prior: 0!
Copyright (C) 2001 by Stefan Matthias Aust (sma@3plus4.de). All rights reserved.

I'm the abstract superclass for special Python runtime objects.!

PyObject subclass: #PyClass
	instanceVariableNames: 'name superclasses suite '
	classVariableNames: ''
	poolDictionaries: ''
	category: 'Sma-Python-Runtime'!

!PyClass commentStamp: 'sma 2/17/2001 22:06' prior: 0!
Copyright (C) 2001 by Stefan Matthias Aust (sma@3plus4.de). All rights reserved.

If I'm grown up, I'll be a Python class :-)!


!PyFunction commentStamp: 'sma 2/17/2001 22:05' prior: 0!
Copyright (C) 2001 by Stefan Matthias Aust (sma@3plus4.de). All rights reserved.

I'm a Python function.!


!PyOperation commentStamp: 'sma 2/17/2001 21:57' prior: 0!
Copyright (C) 2001 by Stefan Matthias Aust (sma@3plus4.de). All rights reserved.

I represent Python operations. Currently, I don't know how to implement oprations but identity-map them onto Smalltalk operation. It is the responsibility of the PyParser who creates me to assign the correct Smalltalk operations.!


!PyScanner commentStamp: 'sma 2/17/2001 22:05' prior: 0!
Copyright (C) 2001 by Stefan Matthias Aust (sma@3plus4.de). All rights reserved.

I'm a simple Python scanner.  I can't deal with line continuations, r-Strings and escape symbols inside strings.  I might have a problem with scanning formfeeds.  Otherwise, I think I can correctly scan Python code and also synthezise indent and dedent tokens. HOWEVER, I do not generate NEWLINE tokens, which is probably a problem. For compatibility with Squeak, my eof mark is #doIt.

Examples:

PyScanner new scanTokens: '
def fact(n):
	if n == 0:
		return 1
	else:
		return fact(n - 1) * n'!


!PyParser commentStamp: 'sma 2/17/2001 22:15' prior: 0!
Copyright (C) 2001 by Stefan Matthias Aust (sma@3plus4.de). All rights reserved.

I'm a simple, still incomplete Python parser based on PyScanner.

Examples:

PyParser new scan: '12' readStream; arithExpr
PyParser new scan: '1+2' readStream; arithExpr
(PyParser new scan: '1+2*3' readStream; arithExpr) eval
PyParser new scan: 'def test(a): 44' readStream; compoundStmt
PyParser new scan: 'def test(a):
	12
	1+2' readStream; compoundStmt
PyParser new scan: 'if a==1: 0' readStream; compoundStmt
(PyParser new scan: 'if 1 > 0: 26' readStream; compoundStmt) eval
PyParser new scan: 'while true:
	1+2
else: 0' readStream; compoundStmt
(PyParser new scan: '{}' readStream) atom
(PyParser new scan: '{12: nil}' readStream) atom
(PyParser new scan: '{12: nil,}' readStream) atom
(PyParser new scan: '{ "A": 1, "B": 2}' readStream) atom

(PyParser new scan: 'class sma: pass' readStream) stmt
(PyParser new scan: 'class sma(object): pass' readStream) stmt
(PyParser new scan: 'class sma(object,thing): pass' readStream) stmt
(PyParser new scan: 'class sma(object,thing):
	def one():
		pass
	def two():
		pass
' readStream) stmt!


!PyStmt commentStamp: 'sma 2/17/2001 22:00' prior: 0!
Copyright (C) 2001 by Stefan Matthias Aust (sma@3plus4.de). All rights reserved.

I represent a Python statement and execute them if asked for. Currently, I only understand if, while and pass.!


!PySuite commentStamp: 'sma 2/17/2001 22:02' prior: 0!
Copyright (C) 2001 by Stefan Matthias Aust (sma@3plus4.de). All rights reserved.

I combine a list of PyStmt statements. When evaluated, I evaluate all my statements and return the result of the evaluation of the last statement.!


!PyVariable commentStamp: 'sma 2/17/2001 22:03' prior: 0!
Copyright (C) 2001 by Stefan Matthias Aust (sma@3plus4.de). All rights reserved.

I represent a variable.!


!PyClass methodsFor: 'accessing' stamp: 'sma 2/17/2001 22:10'!
name
	^ name! !

!PyClass methodsFor: 'accessing' stamp: 'sma 2/17/2001 22:10'!
name: aString
	name _ aString! !

!PyClass methodsFor: 'accessing' stamp: 'sma 2/17/2001 22:10'!
suite
	^ suite! !

!PyClass methodsFor: 'accessing' stamp: 'sma 2/17/2001 22:11'!
suite: anArray
	suite _ anArray! !

!PyClass methodsFor: 'accessing' stamp: 'sma 2/17/2001 22:10'!
superclasses
	^ superclasses! !

!PyClass methodsFor: 'accessing' stamp: 'sma 2/17/2001 22:11'!
superclasses: anArray
	superclasses _ anArray! !


!PyScanner methodsFor: 'expression types' stamp: 'sma 2/17/2001 20:10'!
scanToken
	"Answer the next token. Make sure to synthesize #dedent tokens."

	lastIndent < indent last ifTrue: [^ self dedent].
	^ super scanToken! !

!PyScanner methodsFor: 'multi-character scans' stamp: 'sma 2/17/2001 20:11'!
xComment
	| stopChars stream |
	stopChars _ Array with: Character cr with: Character lf with: 30 asCharacter "doit".
	stream _ WriteStream on: (String new: 200).
	[stopChars includes: hereChar] whileFalse: [stream nextPut: self step].
	currentComment == nil
		ifTrue: [currentComment _ OrderedCollection with: stream contents]
		ifFalse: [currentComment add: stream contents].
	self scanToken! !

!PyScanner methodsFor: 'multi-character scans' stamp: 'sma 2/17/2001 20:12'!
xString
	| delim |
	delim _ self step.
	buffer reset.
	[hereChar = delim
		and: [aheadChar = delim
				ifTrue: [self step. false]
				ifFalse: [true]]]
		whileFalse: 
			[buffer nextPut: self step.
			(hereChar = 30 asCharacter "doit" and: [source atEnd])
				ifTrue: [^self offEnd: 'Unmatched string quote']].
	self step.
	token _ buffer contents.
	tokenType _ #string! !


!PyParser methodsFor: 'parsing definitions' stamp: 'sma 2/17/2001 22:13'!
classDef
	"Rule: classdef: 'class' NAME ['(' testlist ')'] ':' suite"
	"^ <PyClass>"

	| class |
	class _ PyClass new.

	tokenType == #name ifFalse: [self notify: 'class name missing'].
	class name: self advance.

	(self match: #leftParen)
		ifTrue:
			[class superclasses: self testlist.
			self expect: #rightParen].

	self expect: #colon.
	class suite: self suite.
	^ class! !

!PyParser methodsFor: 'parsing definitions' stamp: 'sma 2/17/2001 20:49'!
funcDef
	"Rule: funcdef: 'def' NAME parameters ':' suite"
	"^ <PyFunction>"

	| func |
	func _ PyFunction new.
	tokenType == #name ifFalse: [self notify: 'function name missing'].
	func name: self advance.
	func parameters: self parameters.
	self expect: #colon.
	func suite: self suite.
	^ func! !

!PyParser methodsFor: 'parsing definitions' stamp: 'sma 2/17/2001 21:14'!
lambdaDef
	"Rule: lambdef: 'lambda' [varargslist] ':' test"

	self halt: #todo! !

!PyParser methodsFor: 'parsing definitions' stamp: 'sma 2/17/2001 20:53'!
parameters
	"Rule: parameters: '(' [varargslist] ')'
	 Rule: varargslist: (fpdef ['=' test] ',')*
			('*' NAME [',' ('**'|'*' '*') NAME] | ('**'|'*' '*') NAME)
			| fpdef ['=' test] (',' fpdef ['=' test])* [',']
	 Rule: fpdef: NAME | '(' fplist ')'
	 Rule: fplist: fpdef (',' fpdef)* [',']"
	"^ <Array of: PyNode>"

	| parameters |
	parameters _ OrderedCollection new.
	self expect: #leftParen.
	[self match: #rightParen]
		whileFalse: [
			tokenType == #name ifFalse: [self notify: 'parameter name expected'].
			#todo "implement real arguments".
			parameters addLast: self advance.
			tokenType == #rightParen or: [self expect: #comma]].
	^ parameters asArray! !

!PyParser methodsFor: 'parsing expressions' stamp: 'sma 2/17/2001 21:40'!
andExpr
	"Rule: 
andExpr: shiftExpr ('&' shiftExpr)*"

	| expr |
	expr _ self shiftExpr.
	[self matchToken: #&]
		whileTrue: [
			expr _ PyOperation new
				operation: #bitAnd:;
				arguments: (Array with: expr with: self shiftExpr)].
	^ expr! !

!PyParser methodsFor: 'parsing expressions' stamp: 'sma 2/17/2001 21:32'!
andTest
	"Rule: 
andTest: notTest ('and' notTest)*"

	| expr |
	(expr _ self notTest) ifNil: [^ expr].
	[self matchToken: #and]
		whileTrue: [
			expr _ PyStmt new
				operation: #&;
				arguments: (Array with: expr with: 	self notTest)].
	^ expr! !

!PyParser methodsFor: 'parsing expressions' stamp: 'sma 2/17/2001 21:41'!
arithExpr
	"Rule: 
arithExpr: term (('+'|'-') term)*"

	| expr op |
	expr _ self term.
	[op _ token.
	(self matchToken: #+) or: [self matchToken: #-]]
		whileTrue:
			[expr _ PyOperation new
				operation: op;
				arguments: (Array with: expr with: self term)].
	^ expr! !

!PyParser methodsFor: 'parsing expressions' stamp: 'sma 2/17/2001 21:49'!
atom
	"Rule: 
atom: '(' [testlist] ')' | '[' [testlist] ']' | '{' [dictmaker] '}' | '`' testlist '`' | NAME | NUMBER | STRING+"

	| n |
	(tokenType == #name)
		ifTrue: [^ PyVariable new name: self advance].

	(tokenType == #number)
		ifTrue: [^ PyAtom new atom: self advance].

	(tokenType == #string)
		ifTrue:
			[n _ PyAtom new atom: self advance.
			[tokenType == #string] whileTrue: [n atom: n atom , self advance].
			^ n].

	(self matchToken: #{)
		ifTrue: [^ PyAtom new atom: self dictMaker].

	self halt: #todo.
	^ nil! !

!PyParser methodsFor: 'parsing expressions' stamp: 'sma 2/17/2001 21:45'!
atomTrailer
	"Rule: atomTrailer: atom trailer
	 Rule: trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME
	 Rule: subscriptlist: subscript (',' subscript)* [',']
	 Rule: subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop]
	 Rule: sliceop: ':' [test]"

	"self halt: #todo."
	^ self atom! !

!PyParser methodsFor: 'parsing expressions' stamp: 'sma 2/17/2001 21:38'!
compOp
	"Rule: compOp: '<'|'>'|'=='|'>='|'<='|'<>'|'!!='|'in'|'not' 'in'|'is'|'is' 'not'"

	| i |
	i _ #(< > == >= <= <> !!= in) indexOf: token.
	i > 0
		ifTrue:
			[self advance.
			^ #(< > = >= <= ~= ~= includes:) at: i].

	(self matchToken: #is)
		ifTrue:
			[self error: 'this doesn''t work'.
			(self matchToken: #not) ifTrue: [^ #notKindOf:].
			^ #isKindOf:].
	(self matchToken: #not)
		ifTrue:
			[self error: 'this doesn''t work'.
			(self matchToken: #in)
				ifTrue: [^ #includesNot:]
				ifFalse: [self error: 'need to backup tokenstream here']].
	^ nil! !

!PyParser methodsFor: 'parsing expressions' stamp: 'sma 2/17/2001 21:34'!
comparison
	"Rule: 
comparison: expr (compOp expr)*"

	| expr op |
	(expr _ self expr) ifNil: [^ expr].
	[(op _ self compOp) notNil]
		whileTrue:
			[expr _ PyOperation new
				operation: op;
				arguments: (Array with: expr with: self expr)].
	^ expr! !

!PyParser methodsFor: 'parsing expressions' stamp: 'sma 2/17/2001 21:53'!
dictMaker
	"Rule: 
dictmaker: test ':' test (',' test ':' test)* [',']"
	"^ <Dictionary of: PyNode -> PyNode>"

	| object key value |
	object _ Dictionary new.
	[self match: #rightBrace]
		whileFalse: 
			[key _ self test.
			self expect: #colon.
			value _ self test.
			object at: key put: value.
			((self match: #comma) or: [tokenType == #rightBrace])
				ifFalse: [self notify: ', expected']].
	^ object! !

!PyParser methodsFor: 'parsing expressions' stamp: 'sma 2/17/2001 21:43'!
factor
	"Rule: 
factor: ('+'|'-'|'~') factor | power"

	(self matchToken: #-)
		ifTrue:
			[^ PyOperation new
				operation: #negated;
				arguments: (Array with: self factor)].
	(self matchToken: #~)
		ifTrue:
			[^ PyOperation new
				operation: #bitInvert;
				arguments: (Array with: self factor)].
	self matchToken: #+.
	^ self power! !

!PyParser methodsFor: 'parsing expressions' stamp: 'sma 2/17/2001 21:42'!
notTest
	"Rule: 
notTest: 'not' notTest | comparison"

	(self matchToken: #not)
		ifTrue:
			[^ PyOperation new
				operation: #not;
				arguments: (Array with: self notTest)].
	^ self comparison! !

!PyParser methodsFor: 'parsing expressions' stamp: 'sma 2/17/2001 21:44'!
power
	"Rule: 
power: atom trailer* ('**' factor)*"

	| expr |
	expr _ self atomTrailer.
	[self matchToken: #**]
		whileTrue: [
			expr _ PyStmt new
				operation: #raisedTo:;
				arguments: (Array with: expr with: 	self factor)].
	^ expr! !

!PyParser methodsFor: 'parsing expressions' stamp: 'sma 2/17/2001 21:41'!
shiftExpr
	"Rule: 
shiftExpr: arithExpr (('<<'|'>>') arithExpr)*"
	
	| expr |
	expr _ self arithExpr.
	[#(<< >>) includes: token]
		whileTrue: [
			expr _ PyOperation new
				operation: (self advance == #<< ifTrue: [#bitShift:] ifFalse: [self halt]);
				arguments: (Array with: expr with: self arithExpr)].
	^ expr! !

!PyParser methodsFor: 'parsing expressions' stamp: 'sma 2/17/2001 21:33'!
test
	"Rule: 
test: andTest ('or' andTest)* | lambdef"

	| expr |

	(self matchToken: #lambda) ifTrue: [^ self lambdaDef].

	(expr _ self andTest) ifNil: [^ expr].
	[self matchToken: #or]
		whileTrue: [
			expr _ PyStmt new
				operation: #|;
				arguments: (Array with: expr with: 	self andTest)].
	^ expr! !

!PyParser methodsFor: 'parsing expressions' stamp: 'sma 2/17/2001 21:29'!
testlist
	"Rule: testlist: test (',' test)* [',']"

	| list |
	list _ OrderedCollection with: self test.
	[self matchToken: #,] whileTrue: [list add: self test].
	^list asArray! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:21'!
assertStmt
	"Rule: assertStmt: 'assert' test [',' test]"
	"^ <PyStmt>"

	self halt: #todo! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:21'!
breakStmt
	"Rule: breakStmt: 'break'"
	"^ <PyStmt>"

	self halt: #todo! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:21'!
continueStmt
	"Rule: continueStmt: 'continue'"
	"^ <PyStmt>"

	self halt: #todo! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:21'!
delStmt
	"Rule: delStmt: 'del' exprlist"
	"^ <PyStmt>"

	self halt: #todo! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:21'!
execStmt
	"Rule: execStmt: 'exec' expr ['in' test [',' test]]"
	"^ <PyStmt>"

	self halt: #todo! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:24'!
exprStmt
	"Rule: exprStmt: testlist ('=' testlist)*"
	"^ <PyStmt>"

	"self halt: #todo."
	^ self arithExpr! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:19'!
flowStmt
	"Rule: breakStmt: 'break'"
	"^ <PyStmt>"

	^ PyStmt new operation: #break! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:22'!
forStmt
	"Rule: 
forStmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]"
	"^ <PyStmt>"

	self halt: #todo! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:22'!
globalStmt
	"Rule: globalStmt: 'global' NAME (',' NAME)*"
	"^ <PyStmt>"

	self halt: #todo! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 20:57'!
ifStmt
	"Rule: 
ifStmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]"
	"^ <PyStmt>"

	| stmt st |
	stmt _ PyStmt new
		operation: #if;
		arguments: (Array new: 3).
	stmt arguments at: 1 put: self test.
	self expect: #colon.
	stmt arguments at: 2 put: self suite.
	st _ stmt.
	[self matchToken: #elif]
		whileTrue:
			[st _ st arguments
				at: 3
				put: (PyStmt new
					operation: #if;
					arguments: (Array new: 3)).
			st arguments at: 1 put: self test.
			self expect: #colon.
			st arguments at: 2 put: self suite].
	(self matchToken: #else)
		ifTrue:
			[self expect: #colon.
			st arguments at: 3 put: self suite]
		ifFalse: [st arguments at: 3 put: PyStmt pass].
	^ stmt! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:22'!
importStmt
	"Rule: importStmt: 'import' dottedName (',' dottedName)*
		| 'from' dottedName 'import' ('*' | NAME (',' NAME)*)
	 Rule: dottedName: NAME ('.' NAME)*"
	"^ <PyStmt>"

	self halt: #todo! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:21'!
passStmt
	"Rule: passStmt: 'pass'"
	"^ <PyStmt>"

	^ PyStmt pass! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:22'!
printStmt
	"Rule: printStmt: 'print' (test ',')* [test]"
	"^ <PyStmt>"

	self halt: #todo! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:21'!
raiseStmt
	"Rule: raiseStmt: 'raise' [test [',' test [',' test]]]"
	"^ <PyStmt>"

	self halt: #todo! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:20'!
returnStmt
	"Rule: returnStmt: 'return' [testlist]"
	"^ <PyStmt>"

	self halt: #todo! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 20:52'!
simpleStmt
	"Rule: 
simpleStmt: smallStmt (';' smallStmt)* [';'] NEWLINE"
	"^ <Array of: Stmt>"

	| stmts |
	stmts _ PySuite new add: self smallStmt.
	[self matchToken: #;] whileTrue: [stmts add: self smallStmt].
	^ stmts! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:20'!
smallStmt
	"Rule: 
smallStmt: exprStmt | printStmt | delStmt | passStmt | flowStmt | importStmt | globalStmt | execStmt | assertStmt
	 Rule: flowStmt: breakStmt | continueStmt | returnStmt | raiseStmt"
	"^ <PyStmt>"

	(self matchToken: #print) ifTrue: [^ self printStmt].
	(self matchToken: #del) ifTrue: [^ self delStmt].
	(self matchToken: #pass) ifTrue: [^ self passStmt].
	(self matchToken: #break) ifTrue: [^ self breakStmt].
	(self matchToken: #continue) ifTrue: [^ self continueStmt].
	(self matchToken: #return) ifTrue: [^ self returnStmt].
	(self matchToken: #raise) ifTrue: [^ self raiseStmt].
	(self matchToken: #import) ifTrue: [^ self importStmt].
	(self matchToken: #global) ifTrue: [^ self globalStmt].
	(self matchToken: #exec) ifTrue: [^ self execStmt].
	(self matchToken: #assert) ifTrue: [^ self assertStmt].
	^ self exprStmt! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 20:51'!
stmt
	"
Rule: stmt: simpleStmt | compoundStmt"
	"^ <PyStmt | PySuite>"

	| stmt |
	stmt _ self compoundStmt.
	stmt ifNotNil: [^ stmt].
	^ self simpleStmt! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:22'!
tryStmt
	"Rule: 
tryNtmt: ('try' ':' suite (exceptClause ':' suite)+
			['else' ':' suite] | 'try' ':' suite 'finally' ':' suite)
	 Rule: exceptClause: 'except' [test [',' test]]"
	"^ <PyStmt>"

	self halt: #todo! !

!PyParser methodsFor: 'parsing statements' stamp: 'sma 2/17/2001 21:07'!
whileStmt
	"Rule: 
whileStmt: 'while' test ':' suite ['else' ':' suite]"
	"^ <PyStmt>"

	| stmt |
	stmt _ PyStmt new
		operation: #while;
		arguments: (Array new: 3).
	stmt arguments at: 1 put: self test.
	self expect: #colon.
	stmt arguments at: 2 put: self suite.
	(self matchToken: #else)
		ifTrue:
			[self expect: #colon.
			stmt arguments at: 3 put: self suite]
		ifFalse: [stmt arguments at: 3 put: PyStmt pass].
	^ stmt! !


!PyStmt methodsFor: 'private' stamp: 'sma 2/17/2001 21:13'!
if
	^ arguments first eval
		ifTrue: [arguments second eval]
		ifFalse: [arguments third eval]! !


!PyStmt class methodsFor: 'instance creation' stamp: 'sma 2/17/2001 20:57'!
pass
	^ self new operation: #pass! !


!PyStmt class reorganize!
('instance creation' pass)
!

PyStmt removeSelector: #break!
PyStmt removeSelector: #stmtBreak!

!PyParser reorganize!
('parsing definitions' classDef funcDef lambdaDef parameters suite)
('parsing expressions' andExpr andTest arithExpr atom atomTrailer compOp comparison dictMaker expr factor notTest power shiftExpr term test testlist xorExpr)
('parsing statements' assertStmt breakStmt compoundStmt continueStmt delStmt execStmt exprStmt flowStmt forStmt globalStmt ifStmt importStmt passStmt printStmt raiseStmt returnStmt simpleStmt smallStmt stmt tryStmt whileStmt)
('private' expect: expectToken: match: matchToken:)
!


!PyClass reorganize!
('accessing' name name: suite suite: superclasses superclasses:)
!