Opened 3 months ago

Last modified 2 weeks ago

#5819 new task

Python 3 compatibility

Reported by: randersson Owned by: Jesper Mattsson
Priority: major Milestone:
Component: Build Version: trunk
Keywords: Cc:

Description

A ticket to collect all changes required to make JModelica compliant with Python 3.

Change History (32)

comment:1 Changed 3 months ago by randersson

r13322 Branched JModelica

comment:2 Changed 2 months ago by randersson

r13461 Performed 2to3 on the entire src/Python directory in order to make the code compatible. I've not reviewed all changes while doing this commit but will do while starting to run the tests. I'm not sure everything is converted correctly, for example the change in test_casadi_collocation row 1610.

comment:3 Changed 2 months ago by randersson

r13466 Fixed issues with metaclasses from 2to3 changes
r13468 Added back licensing text that was apparently removed for some unknown reason in commit r13466
r13469 Added correct tabs into pyjmi example, since apparently this was removed at some unknown time, but looking at trunk it is clear that this change is the desired one.

comment:4 Changed 2 months ago by randersson

r13483 Updated tests with error messages in order to debug what is wrong with the tests properly since they are simply failing now with AssertionError. I did also replace some asserts from other python packages with the standard assert
r13484 Added changes from running 2to3 on core.py and xmlparser.py directly to PyFMI trunk

comment:5 Changed 2 months ago by randersson

r13494 Adding conditional expression into constructors of Comment and Node to test if py2 and py3 compatibility.
r13495 Testing by simply removing the asserts and updating a conditional changing the type of input to list.
r13496 Added fix for KeepLastStream with attribute close on branch for python3 development since the error seems to exist even here.

comment:6 Changed 2 months ago by randersson

r13505 Added fix for remaining 13 failing tests

comment:7 Changed 2 months ago by randersson

r13510 Fixed such that build system defaults to 64bit in case user does no longer choose to build both 32bit and 64bit for python. Note that changes in configure and Makefile.in are automatically generated solely by changes in configure.ac and Makefile.am

comment:8 Changed 2 months ago by randersson

r13513 Updated expression in configure.ac since the right bracket is interpreted as a command and not as an evaluation of the variable.
r13514 Also added the change in configure that should have been added in r13513

comment:9 Changed 2 months ago by randersson

r13515 Replaced tabs with spaces because they are causing TabError in Python 3.
r13517 Replaced previous split of using module string to now use the class str. Reason is using split with module string is not compatible with python 3.
r13522 Removed keyword argument since it is not needed and is actually called meta instead of metaclass.
r13523 Changed to asserts to use standard assert. Replaced assertTrue with assert_ again which was previously changed by script 2to3. It appears that assertTrue is a non-existent function, thus this original change by 2to3 is a mystery.

comment:10 Changed 2 months ago by randersson

r13535 Correcting suffix for python dependencies for the unix build system
r13536 Also applied same changes to Makefile.in that should have been with r13535
r13537 Testing changing the exceptions caught from JavaException to JException since Java Throwable is converted to JException since 0.7.0 (JavaException does not exist in jpype 0.7.0). Also changed type of keys when constructing problem nodes since xmlparser have issues with keys being of type str in py3. These changes are not tested yet with py2.
r13543 Updated init.py to point at correct directory when running python3
r13544 Fixed hopefully final issue with compile_fmu returning None every time even during succesful compilations. Appears that an entry to dictionary was not raising any error even though type of key was incorrect.

comment:11 Changed 2 months ago by randersson

r13566 Testing if the exceptions from jpype is now correctly raised using both versions of jpype (in py2 and py3). This commit contains some #... debug information that will be removed later.

comment:12 Changed 2 months ago by randersson

r13574 Updated test to no longer use nose and use numpy instead due to bug with nose calling function round with invalid data types. This error does not occur with python 2 but does with python 3. Also updated all assert a == b with a proper error message in case test fails in future.

comment:13 Changed 2 months ago by randersson

r13579 Changed externals to pyfmi trunk in order to test if tests with parsing of logs are OK. Will go back to a tag or branch later.

r13583 Updates to code after review comments. Now all the 'is python 3' flags are imported from pyfmi.

r13584 Fixed imports

r13590 Fixed imports now it is properly working, also got rid of import of version_info

r13591 Fixed typo of import to from

r13592 Hopefully fixed issue with text encoding causing parse_jmi_log to to return objects of type bytes instead of the str.

r13593 Updated interface to use the java method setTargetPlatforms(...) instead of directly modifying the field targetPlatforms. This was earlier possible to do due to a bug in jpype 0.6 but seems to have been fixed in jpype 0.7.0. This can easily be tested locally by trying to access property targetPlatforms through the python object ModelicaCompiler.

comment:14 Changed 8 weeks ago by randersson

r13596 Updated parser to simply skip all the empty elements, instead of skipping every element with even index which seemed to have been empty previously anyway. Also convert all elements to str in order to avoid any differences between python2 and python3 causing unicode/str/bytes in different cases.
r13597 Updated class method to properly convert the format of strings to strings since there is currently a discrepancy between py2 and py3. In py2 its u'text' instead of simply just 'text' as in py3.
r13600 Merged JM trunk into branch.

comment:15 Changed 7 weeks ago by randersson

r13618 Replaced IOError to OSError since it has been merged since python 3.3. Fixed such that self.last is no longer decoded from bytes and is instead the decoding + convert to str is only done on the objects where it is needed (such as when using count, find, split). This enables self.last to be of the type required by the version of xmlparser used, and no longer overridden.
r13619 Testing if the following fix enables us to remove the conversion of keys to bytes. Also added the import of py3 flag in compiler_logging

comment:16 Changed 7 weeks ago by randersson

r13626 Added test for test imports of assimulo solvers
r13627 Disabled test on branch meanwhile because assimulo is currently not included
r13634 Updated two additional IOErrors to OSErrors, see r13618
r13635 Updated set_target_platforms on trunk with half of the fix from the py3 branch, see similar change in commit r13593.

comment:17 Changed 7 weeks ago by randersson

r13641 Moved the conversion of type of the keys to parser.py -> startElement(...) because it is the entry point for xml package. This change is based on the difference between python2 and python3, where the keys in python2 would be added with type unicode, while python3 adds the keys with type str.
r13642 Added another test for non existant files.
r13643 Updated test with better error message
r13644 Merged r13635 to release branch.
r13648 Added type() around exception in test in order to get the correct error message
r13649 Updated test with platform specific message since apparently the error message thrown from java is different on linux and windows.
r13651 Merged trunk into branch
r13654 Modified condition with sys.platform since CentOS 7.4 apparently returns 'linux2' when using sys.platform

comment:18 Changed 7 weeks ago by randersson

r13656 Testing if overriding the variables for CasADi 32-bit with 64-bit values enables build of CasADi with 64-bit only

comment:19 Changed 7 weeks ago by randersson

r13662 Adding automated changes from Makefile.am to Makefile.in, related to r13656

comment:20 Changed 6 weeks ago by randersson

r13663 Testing if changing the path to libpythonXY.a resolves the issue with numpy build errors. Note that these changes are obviously not final and are just for testing.
r13667 Adding echo in order to see if variables are actually overriden or not
r13668 Also adding autogenerated change from previous commit
r13678 Updated time period in copyright information
r13680 Changed svn external to new casadi branch
r13684 Added additional echos in order to see if variables are changed during runtime before build casadi.

comment:21 Changed 6 weeks ago by randersson

r13678 Updated time period in copyright information
r13680 Changed svn external to new casadi branch
r13684 Added additional echos in order to see if variables are changed during runtime before build casadi.
r13685 Adding additional echo and creating a new layer in order to properly test change of variables for building only the 64bit, this will no longer cause error messages with variables that are changed during build since we now add the TMP_..., also note that this fix is far from final and is only a test to see if the correct gcc is used by doing this

Last edited 6 weeks ago by randersson (previous) (diff)

comment:22 Changed 6 weeks ago by randersson

r13691 Updated additional variables that were not updated previously. In the current state casadi is building all the way until ifcasadi target, it currently fails due to issues with locating DLL and I believe this is because of missing updates of variables that has been done now in this change.
r13693 Updated some additional variables that were missing the TMP_ prefix.
r13694 Updated CMakeLists.txt to be python 3 compatible
r13698 Replaced toString(...) with toPyUnicode(...) in order to test if this resolve the issue with building ModelicaCasADiInterface using a JCC compatible with Python 3. The switch to toPyUnicode is based on using diff on JCCEnv.h from the two different jcc versions. Note that this change is not Python 2 compatible, but, is for testing if it resolves the current build issue.

comment:23 Changed 6 weeks ago by randersson

r13706 Testing by adding a preprocessor since there are issues loading the instance method toPyUnicode

comment:24 Changed 5 weeks ago by randersson

r13724 Added new method to convert jobject to strings. Also replaced previous test of "toPyUnicode" with this method since the test was wrong. Note that one toString() is still in place and it should be since it is the original toString function. At this point modelicacasadiinterface is building.
r13740 Updated assimulo external to point at branch
r13745 Adding echo in order to debug segfaults with jm_tests
r13749 Fixed such that the build system identifies correct python libs depending on what type of JModelica that is being built.

comment:25 Changed 5 weeks ago by aramle

r13752 Adding CMakeLists.txt for SuperLU. Works to build with MSVS.

comment:26 Changed 5 weeks ago by aramle

r13753 Adding target install_cmake to superlu makefile. Adding config file to build with MSVS 2015.

comment:27 Changed 5 weeks ago by randersson

r13754 Updated paths for previous generation of JM to be correct. Note these are not used when build with python 3 anyway.
r13757 Added missing prefix SUPERLU_ADDON for sundials
r13758 Merged r13757 to trunk
r13759 Adding more echo to jm_tests
r13770 Added more echos
r13775 Replaced updating class object dict with setattr because the previous way did not work in Python 3. Also updated test to use numpy asserts to get proper error messages
r13778 Updated Additional_versions in casadi makefiles
r13779 Testing setting WITH_OPENMP to OFF

comment:28 Changed 4 weeks ago by randersson

r13749 Fixed such that the build system identifies correct python libs depending on what type of JModelica that is being built.
r13750 Added more echo in order to debug jm_tests segfaults
r13752 Adding CMakeLists.txt for SuperLU. Works to build with MSVS.
r13753 Adding target install_cmake to superlu makefile. Adding config file to build with MSVS 2015.
r13754 Updated paths for previous generation of JM to be correct. Note these are not used when build with python 3 anyway.
r13757 Added missing prefix SUPERLU_ADDON for sundials
r13759 Adding more echo to jm_tests
r13760 Fixing namings for superlu mt in CMakeLists.txt
r13761 Not yet working for Sundials build with MSVS
r13762 Adding one missed header file to install.
r13763 Fixing so that Sundials can link with MSVS built SuperLU.
r13764 Removing debug flag in sundials makefile
r13765 Renamed configure file
r13766 Updating to build MSVS on Jenkins.
r13767 Renamed targets to fit better with Jenkins.
r13768 Made sure to remove install dir when cleaning and correcting typo.
r13769 Adding support on Jenkins for msvs make target.
r13770 Added more echos
r13772 Preparations for building Assimulo with Intel fortran and msvs.
r13774 Fixing so that Sundials can link with MSVS built SuperLU (minor additional fix)
r13775 Replaced updating class object dict with setattr because the previous way did not work in Python 3. Also updated test to use numpy asserts to get proper error messages
r13778 Updated Additional_versions in casadi makefiles
r13779 Testing setting WITH_OPENMP to OFF
r13780 Making namings consistent to work with Jenkins.
r13782 Switched back to other CasADi branch
r13783 Switched branch PyFMI external from trunk to tag 2.5.5.
r13785 Updating makefile and default_config for building assimulo
r13788 Updating Python makefile to accomodate for external resources being built with win64 now.
r13791 Replaced all nose.tools.assert_almost_equal with similar method from numpy instead. Reason is that nosetests seems to fail due to bug:
r13792 Adding variable to be able to define Python base dir.
r13794 Modified some assert a == b to also follow up with a proper error message when the assert fails.

comment:29 Changed 4 weeks ago by randersson

r13795 Adding variable to be able to define Python base dir and updating assimulo test to work with Python 3.
r13796 Adding variable to be able to define Python base dir in more locations.
r13797 Updating the FMU that is generated for the model Modelica.Mechanics.Rotational.Examples.CoupledClutches and which has generationTool changed to "Unknown Tool" in modelDescription.xml.
r13799 Removed old echo "..."
r13800 Merged trunk into branch
r13801 Removed keyword 'places' because it was already in 3rd order but with numpy it is called decimal
r13802 Fixed such that assimulo uses assimulo makefile instead of directly calling setup.py
r13803 Also adding echo "..." in order to verify that overriding the variable works fine.
r13804 Changed from A to a
r13805 Set PYTHONHOME in unix config to empty
r13806 Updated typo with SUPERLU_HOME for unix systems
r13807 Making sure install directory exists.
r13808 Modified PYTHONHOME in Assimulo configs
r13809 Updated the unix config since it using an empty variable before leading to incorrect path
r13810 Updated the superlu paths since there are no superlu_install directories
r13812 Removed removal of directories from build targets and put them under clean instead
r13813 Fixed tab in makefile
r13819 Updated relative tolerance in test from 1e-7 (default) to 1e-6 due to updates in both python and assimulo
r13820 Changed PyFMI external to trunk
r13821 Added more echo to jm_tests
r13824 Updating file to open to be Python 3 compatible.
r13827 Increasing memory available for ant.
r13833 Updated ant_tests() in jm_tests to use the variable ANT_OPTS.

comment:30 Changed 4 weeks ago by randersson

r13833 Updated ant_tests() in jm_tests to use the variable ANT_OPTS.
r13839 Replaced e.message with str(e) for Python 3
r13857 Added echo of jm_tests
r13859 More debug output to jm_tests.in
r13862 More debug output to jm_tests.in

comment:31 Changed 3 weeks ago by randersson

r13865 More debug output to jm_tests.in
r13867 Adding a retrying fallback in case reading of args fails.
r13872 Testing parsing arguments without sed.
r13873 Testing parsing arguments without sed.
r13881 Removing ant_opts since that is not an argument to ant but an environment variable.
r13888 Testing outcommenting sed to see if it improves the behavior.
r13891 Correcting typo.
r13893 Correcting typo.
r13895 Testing without mktemp
r13902 Updates after review.
r13903 Testing without mktemp continuing.
r13904 Replaced "...." with the correct type of delimiter
r13905 Fixed another tab issue in Makefile.am
r13909 Reverted r13092-r13095 because it seems that number of make processes escalates

comment:32 Changed 2 weeks ago by randersson

r13909 Reverted r13092-r13095 because it seems that number of make processes escalates
r13915 Added back r13092 without including with the changes in the Makefile, for some mysterious reason it seems to loop itself and rebuild the same target over and over. Iniitally it was thought that it looped because it missed a cd ... but it appears while trying locally it still looped. Instead of wasting more time on the minor modification we stick with this change
r13929 Updated error message in test in case of failure
r13942 Updating information on Python dependencies (outcommented so that it is still not yet written by default.
r13943 Merged trunk to branch
r13951 Updated test to get version through the method get_version and types_platform through the class attribute instead of the method _get...
r13956 Updating groovy for assimulo build.
r13957 Adding extra bat
r13965 Removed empty line causing target to end build

Note: See TracTickets for help on using tickets.