(pheSrSSKrSSKrSSKrSSKrSSKrSSKJr SSKrSSK r SSK r SSK JrJr SSKJrJrJrJrJrJrJrJrJr SSKJr SSKJrJrJrJ r J!r!J"r"J#r#J$r$J%r%J&r&J'r'J(r(J)r)J*r*J+r+J,r,J-r-J.r. SS K/J0r0 S r1S r2S r3S r4"SS\5r5Sr6"SS5r7Sr8\\SSr9\\SSr:\\SSr;\\SSr<"SS5r="SS5r>g)a; Classes for read / write of matlab (TM) 5 files The matfile specification last found here: https://www.mathworks.com/access/helpdesk/help/pdf_doc/matlab/matfile_format.pdf (as of December 5 2008) ================================= Note on functions and mat files ================================= The document above does not give any hints as to the storage of matlab function handles, or anonymous function handles. I had, therefore, to guess the format of matlab arrays of ``mxFUNCTION_CLASS`` and ``mxOPAQUE_CLASS`` by looking at example mat files. ``mxFUNCTION_CLASS`` stores all types of matlab functions. It seems to contain a struct matrix with a set pattern of fields. For anonymous functions, a sub-fields of one of these fields seems to contain the well-named ``mxOPAQUE_CLASS``. This seems to contain: * array flags as for any matlab matrix * 3 int8 strings * a matrix It seems that whenever the mat file contains a ``mxOPAQUE_CLASS`` instance, there is also an un-named matrix (name == '') at the end of the mat file. I'll call this the ``__function_workspace__`` matrix. When I saved two anonymous functions in a mat file, or appended another anonymous function to the mat file, there was still only one ``__function_workspace__`` un-named matrix at the end, but larger than that for a mat file with a single anonymous function, suggesting that the workspaces for the two functions had been merged. The ``__function_workspace__`` matrix appears to be of double class (``mxCLASS_DOUBLE``), but stored as uint8, the memory for which is in the format of a mini .mat file, without the first 124 bytes of the file header (the description and the subsystem_offset), but with the version U2 bytes, and the S2 endian test bytes. There follow 4 zero bytes, presumably for 8 byte padding, and then a series of ``miMATRIX`` entries, as in a standard mat file. The ``miMATRIX`` entries appear to be series of un-named (name == '') matrices, and may also contain arrays of this same mini-mat format. I guess that: * saving an anonymous function back to a mat file will need the associated ``__function_workspace__`` matrix saved as well for the anonymous function to work correctly. * appending to a mat file that has a ``__function_workspace__`` would involve first pulling off this workspace, appending, checking whether there were any more anonymous functions appended, and then somehow merging the relevant workspaces, and saving at the end of the mat file. The mat files I was playing with are in ``tests/data``: * sqr.mat * parabola.mat * some_functions.mat See ``tests/test_mio.py:test_mio_funcs.py`` for the debugging script I was working with. Small fragments of current code adapted from matfile.py by Heiko Henkelmann; parts of the code for simplify_cells=True adapted from http://blog.nephics.com/2019/08/28/better-loadmat-for-scipy/. N)BytesIO) native_code swapped_code) MatFileReader docfillermatdims read_dtype arr_to_charsarr_dtype_number MatWriteError MatReadErrorMatReadWarning) VarReader5) MatlabObjectMatlabFunctionMDTYPES NP_TO_MTYPES NP_TO_MXTYPES miCOMPRESSEDmiMATRIXmiINT8miUTF8miUINT32 mxCELL_CLASSmxSTRUCT_CLASSmxOBJECT_CLASS mxCHAR_CLASSmxSPARSE_CLASSmxDOUBLE_CLASS mclass_info mat_struct)ZlibInputStreamc[U[R5=(a? URS:=(a) URS:=(a [US[ 5$)zBDetermine if elem is an array and if first array item is a struct.r) isinstancenpndarraysizendimr")elems H/var/www/html/venv/lib/python3.13/site-packages/scipy/io/matlab/_mio5.py _has_structr,isF tRZZ ( ,dii!m ,$))a- , tAw +-c/nUHqn[U[5(aUR[U55 M4[ U5(aUR[ U55 M`URU5 Ms U$)zuConstruct lists from cell arrays (loaded as numpy ndarrays), recursing into items if they contain mat_struct objects.)r%r"append_matstruct_to_dictr,_inspect_cell_array)r' elem_listsub_elems r+r1r1osiI h + +   /9 :  " "   0: ;   X &  r-c0nURHYnURUn[U[5(a[ U5X'M6[ U5(a[ U5X'MUX1U'M[ U$)z/Construct nested dicts from mat_struct objects.) _fieldnames__dict__r%r"r0r,r1)matobjdfr*s r+r0r0}sc A   q! dJ ' '%d+AD   &t,ADaD  Hr-cUHNn[X[5(a[X5X'M+[X5(dM?[ X5X'MP U$)z,Convert mat objects in dict to nested dicts.)r%r"r0r,r1)r8keys r+_simplify_cellsr<sK afj ) )'/AF  (0AF  Hr-c~^\rSrSrSr\S U4Sjj5rSrSrSr Sr S Sjr SS jr S r S rU=r$)MatFile5ReaderaReader for Mat 5 mat files Adds the following attribute to base class uint16_codec - char codec to use for uint16 char arrays (defaults to system default codec) Uses variable reader that has the following standard interface (see abstract class in ``miobase``:: __init__(self, file_reader) read_header(self) array_from_header(self) and added interface:: set_stream(self, stream) read_full_tag(self) c >[T U]UUUUUUUUU 5 U (d[R"5n XlSUlSUlg)zInitializer for matlab 5 file format reader %(matstream_arg)s %(load_args)s %(struct_arg)s uint16_codec : {None, string} Set codec to use for uint16 char arrays (e.g., 'utf-8'). Use system default codec if None N)super__init__sysgetdefaultencoding uint16_codec _file_reader_matrix_reader) self mat_stream byte_order mat_dtype squeeze_mechars_as_stringsmatlab_compatiblestruct_as_record verify_compressed_data_integrityrEsimplify_cells __class__s r+rBMatFile5Reader.__init__sV*         ,  113L( "r-cURRS5 URRS5nURRS5 US:H=(a S=(d S$)z*Guess byte order. Sets stream pointer to 0~rsIM<>)rIseekread)rHmis r+guess_byte_orderMatFile5Reader.guess_byte_ordersP S! __ ! !! $ QU{"s)c)r-c0n[URSSn[URU5nUSR 5R S5US'USS- nUSS-nS XE4-US 'U$) zRead in mat 5 file header dtypes file_header descriptions __header__versionz%d.%d __version__)rrJr rIitemstrip)rHhdict hdr_dtypehdrv_majorv_minors r+read_file_headerMatFile5Reader.read_file_headersDOO,X6}E )4!-0557==lKli.A%i.4'&');;m r-cD[U5Ul[U5Ulg)zPRun when beginning read of variables Sets up readers from parameters in `self` N)rrFrGrHs r+initialize_readMatFile5Reader.initialize_reads't,(.r-cURR5upUS:d [S5eURR 5U-nU[ :XaZ[ URU5nURRU5 URnURR5upO'SnURRUR5 U[:Xd[SU-5eURRU5nXc4$)a;Read header, return header, next position Header has to define at least .name and .is_global Parameters ---------- None Returns ------- header : object object that can be passed to self.read_var_array, and that has attributes .name and .is_global next_position : int position in stream of next variable rzDid not read any bytesFz$Expecting miMATRIX type here, got %d) rF read_full_tag ValueErrorrItellrr#rG set_streamrPr TypeError read_header)rHmdtype byte_countnext_posstreamcheck_stream_limitheaders r+read_var_headerMatFile5Reader.read_var_headers""..<<>A~56 6??'')J6 \ !$T__jAF    * *6 2!%!F!F !%!4!4!B!B!D FJ!&     * *4?? ;!BVKL L$$001CDr-c8URRX5$)aLRead array, given `header` Parameters ---------- header : header object object with fields defining variable header process : {True, False} bool, optional If True, apply recursive post-processing during loading of array. Returns ------- arr : array array with post-processing applied or not according to `process`. )rGarray_from_header)rHrprocesss r+read_var_arrayMatFile5Reader.read_var_arrays"""44VEEr-c[U[5(aU/nOUb [U5nURR S5 UR 5 UR 5n/US'UR5(Gd%UR5up4URcSOURRS5nXR;a SUS3n[R"U[SS 9 US :XaS nS nOS nUb"XQ;aURR U5 MURX75nURR U5 XU'UR"(aUSR%U5 Ub!UR'U5 [)U5S:XaOUR5(dGM%UR*(a [-U5$U$![a0n [R"SUSU S3[ SS 9 SU 3nSn A NSn A ff=f)zget variables from stream as dictionary variable_names - optional list of variable names to get If variable_names is None, then get all variables in file Nr __globals__Nonelatin1zDuplicate variable name "z{" in stream - replacing previous with new Considerscipy.io.matlab.varmats_from_mat to split file into single variable filesrV) stacklevel__function_workspace__FTzUnreadable variable "z ", because ""z Read error: )r%strlistrIrYrrrn end_of_streamrnamedecodewarningswarnrrrWarning is_globalr/removelenrQr<) rHvariable_namesmdictrk next_positionrmsgrreserrs r+ get_variablesMatFile5Reader.get_variables%s nc * *,-N  '!.1N Q %%'!m$$&&!%!5!5!7 C XX-6388??83LD}/v666  c>a@rz/ )d.H$$]3 +))#7 OO  /$K}}m$++D1)%%d+~&!+I$$&&J   "5) )L!  + +D6cU!D+%SE*  +sG G; &G66G;c\URRS5 UR5 UR5 /nUR 5(dUR 5up#UR cSOUR RS5nUS:XaSnURRU5nUR(aSnO![R"URS5nURXEU45 URRU5 UR 5(dMU$)zlist variables from stream rrrrrlogicalunknown)rIrYrrrnrrrrrGshape_from_header is_logicalr!getmclassr/)rHvarsrkrrshapeinfos r+list_variablesMatFile5Reader.list_variables`s Q  $$&&!%!5!5!7 C XX-6388??83LDrz/''99#>E~~ "szz9= KKd+ , OO  /$$&& r-)rFrGrE) NFFTFTTNF)TN)__name__ __module__ __qualname____firstlineno____doc__rrBr\rnrrrrrr__static_attributes__ __classcell__)rRs@r+r>r>se&! !"&#("&26" %$#$#L*  /! FF&9vr-r>c[U5nURS5 [[SSRnUR U5nURS5 UR 5 UR5 UR5n/nUR5(dUnUR5uptURcSOURRS5nURU5 XF- n UR U 5n [5n U RU5 U RU 5 U RS5 URX45 UR5(dMU$)aPull variables out of mat 5 file as a sequence of mat file objects This can be useful with a difficult mat file, containing unreadable variables. This routine pulls the variables out in raw form and puts them, unread, back into a file stream for saving or reading. Another use is the pathological case where there is more than one variable of the same name in the file; this routine returns the duplicates, whereas the standard reader will overwrite duplicates in the returned dictionary. The file pointer in `file_obj` will be undefined. File pointers for the returned file-like objects are set at 0. Parameters ---------- file_obj : file-like file object containing mat file Returns ------- named_mats : list list contains tuples of (name, BytesIO) where BytesIO is a file-like object containing mat file contents as for a single variable. The BytesIO contains a string with the original header and a single var. If ``var_file_obj`` is an individual BytesIO instance, then save as a mat file with something like ``open('test.mat', 'wb').write(var_file_obj.read())`` Examples -------- >>> import scipy.io >>> import numpy as np >>> from io import BytesIO >>> from scipy.io.matlab._mio5 import varmats_from_mat >>> mat_fileobj = BytesIO() >>> scipy.io.savemat(mat_fileobj, {'b': np.arange(10), 'a': 'a string'}) >>> varmats = varmats_from_mat(mat_fileobj) >>> sorted([name for name, str_obj in varmats]) ['a', 'b'] rr_r`rr)r>rYrritemsizerZrrrnrwrrrrrwriter/) file_objrdrhdr_lenraw_hdrr named_matsstart_positionrkrr|var_strout_objs r+varmats_from_matrys.P  "C MM!k"8,];DDGmmG$G MM!MMOMJ!!& 002)vsxxx/H n%"3 -- +) g g Q4/*!! r-c\rSrSrSrSrg)EmptyStructMarkeriz!#))--"34F 99  , .,,.%%cii&<&<&>?CXXcll* ?  ( (j A  & &sJ ?r-c[R"S[5nUS-U-US'URSS9US'UR U5 g)Nrbyte_count_mdtyperrdata)r&zeros NDT_TAG_SMALLrr)rHrr{r|tags r+r"VarWriter5.write_smalldata_element!sIhhr=)$."$4#>  kkk,F  r-c[R"S[5nX$S'X4S'URU5 URU5 US-nU(a"URR SSU- -5 gg)Nrr{r|rd)r&r NDT_TAG_FULLrrr)rHrr{r|r bc_mod_8s r+r VarWriter5.write_regular_element)skhhr<(H &L  >     " "7aj#9 : r-cRURnURnURR5UlUR UR 5 [R"S[5n[US'SUS'US-US--US--n X)S--US'XXS 'UR U5 UR[R"US S 95 [R"U5nUS :XaURU[S 5 OURU[5 S UlSUlg)aRWrite header for given data options shape : sequence array shape mclass - mat5 matrix class is_complex - True if matrix is complex is_logical - True if matrix is logical nzmax - max non zero elements for sparse arrays We get the name and the global flag from the object, and reset them to defaults after we've used them r data_typerdr|rVr flags_classnzmaxi4rrrFN)rrrrw _mat_tag_posrmat_tagr&rNDT_ARRAY_FLAGSrrrrrr) rHrr is_complexrrrrafflagss r+ write_headerVarWriter5.write_header5s$~~''  ,,113 & XXb/ *";<a)q.0:?B"aZ/=7   288E67zz$ 2:  ( (vq 9   tV ,#r-c&URR5nURRU5 X!- S- nUS:a [S5eX0RS'UR UR5 URRU5 g)Nrdlz-Matrix too large to save with Matlab 5 formatr|)rrwrYr rr)rH start_poscurr_posr|s r+update_matrix_tagVarWriter5.update_matrix_tag`s##((* i()A-  !+, ,%/ \" & h'r-c>X0lX lURU5 g)a7Write variable at top level of mat file Parameters ---------- arr : array_like array-like object to create writer for name : str, optional name as it will appear in matlab workspace default is empty string is_global : {False, True}, optional whether variable will be global on load into matlab N)rrr)rHrrrs r+ write_topVarWriter5.write_topks( 3r-cVURR5n[RR U5(a#UR U5 UR U5 g[U5nUc[SUS[U5S35e[U[5(aURU5 O[U[5(a [S5eU[LaUR!5 OUR"R$(aUR'U5 OUR"R((aUR+U5 OSUR"R,S;a(UR.(aSnOSnUR1X45 OUR3U5 UR U5 g) zWrite `arr` to stream at top and sub levels Parameters ---------- arr : array_like array-like object to create writer for NzCould not convert z (type z ) to arrayzCannot write matlab functions)USUTF8ascii)rrwscipysparseissparse write_sparser"rryrr%r write_objectrr rwrite_empty_structrfields write_struct hasobject write_cellskindr write_char write_numeric)rHr mat_tag_posrcodecs r+rVarWriter5.writesD&&++- <<  % %   c "  " "; / C  <0WT#YKzRS S dL ) )   d # n - - ?@ @ & &  # # % ZZ     d # ZZ ! !   T " ZZ__ *## OOD (   t $ {+r-cDURRS:HnURRS:Hn[URRSSnUR[XR5UUUS9 U(a7URUR5 URUR5 gURU5 g![aL U(aUR S5nO*U(aUR S5nOUR S5n[ nNf=f)Ncbrc128i1f8)rr) rr6rrKeyErrorastyper rr rrrealimag)rHrimagflogifrs r+r8VarWriter5.write_numerics #% #% $"399==#45F '#||4 %*%*  ,    sxx (   sxx (   s #% $jj(jj&jj&#F $s C ADDcURS:Xd[R"US:H5(aSS[R"URS/5-nUR U[ 5 URU[S5 g[U5nURnUR U[ 5 URRS:XaUR(a[R"U5n[R"S[!X5UR"R%5S9nUR'5R)U5n[R"[+U54S US9nUR-U[S 9 g) z4Write string array `arr` with given `codec` rr)rrVNr(rrrbufferS1r{)r(r&allmaxr)rrrrr rrr6mathprodr'r Tcopyrgencoderr)rHrr:rn_charsst_arrsts r+r7VarWriter5.write_chars 88q=BFF3"9--266388Q-00E   e\ 2  ( (fa 8 3  %. 99>>S SXX ii&GZZb&6s&D'*uuzz|5F%%e,B**CG:#'$&(C 3v.r-cpUR5nUR5 URRS:HnURRS:HnURnUR [ XR5[UUUS:XaSOUS9 URURRS55 URURRS55 URURR5 U(a&URURR5 gg)zSparse matrices are 2D r=r>rr)rrrrN)tocsc sort_indicesrr6nnzrr rrrindicesrCindptrrrDrE)rHrArrnzs r+r/VarWriter5.write_sparses IIK ggllc) ggllc) UU '#||4(%/%/%'1W"  6 199++D12 188??401 166;;'    qvv{{ + r-cUR[XR5[5 [R "U5R S5nUHnURU5 M g)Nr)rr rrr& atleast_2dflattenr)rHrr_els r+r5VarWriter5.write_cellssN '#||4& ( MM#  & &s +B JJrNr-cURS[5 UR[R"S[R S95 UR[R"/[R S95 g)N)rrrr)rrrr&rint32int8rqs r+r1VarWriter5.write_empty_structsL &.1 288ARXX67 288Bbgg67r-cxUR[XR5[5 UR U5 gr)rr rr _write_itemsrs r+r3VarWriter5.write_struct s, '#||4( * #r-cRURRVs/sHo"SPM nn[UVs/sHn[U5PM sn5S-nUR=(a S=(d SnXV:a[ SUS- -5eUR [R"U/SS95 UR [R"USU-S9[S 9 [R"U5RS 5nUHnUHnURX5 M M! gs snfs snf) Nrr@ z+Field names are restricted to %d charactersrrzS%drMr) rdescrrOrrrvrr&rrrcrdr) rHrr9 fieldnames fieldnamelength max_lengthr_res r+rlVarWriter5._write_itemss$'IIOO4OqdO 4jAjc)njAB1D++29r  J(l,- - 288VHD9:  HHZu'7 8   MM#  & &s +B 25! 5As DD$cUR[XR5[5 UR [ R "URSS9[S9 URU5 g)z]Same as writing structs, except different mx class, and extra classname element after header r)rrMN) rr rrrr&r classnamerrlrs r+r0VarWriter5.write_object#sU '#||4( * 288CMM="(  * #r-)rrrrrrrr)FFr)r+)rrrrrr&rrrrrBrrrrrrr"r%rr8r7r/r5r1r3rlr0rrr-r+rrs/hhr<(G GH$7" @ ;!& % )$V ((%,N$2)/V,(8 ""r-rcF\rSrSrSr\SSj5rSrS SjrSr g) MatFile5Writeri.zClass for writing mat5 files NcxXlX lX0lU(aX@lO/UlXPlX`lSUlg)zInitialize writer for matlab 5 format files Parameters ---------- %(do_compression)s %(unicode_strings)s global_vars : None or sequence of strings, optional Names of variables to be marked as global for matlab %(long_fields)s %(oned_as)s N)rdo_compressionr global_varsrr_matrix_writer)rHrr}rr~rrs r+rBMatFile5Writer.__init__1s:$',. * !D  0 "r-cH[R"S[5nS[RS[ R "53US'SUS'[R"SS[R"S5S 9US 'URRUR55 g) NrzMATLAB 5.0 MAT-file Platform: z, Created on: rarcS2iIMrJ endian_test) r&r NDT_FILE_HDRosrtimeasctimer'uint16rrr)rHrks r+write_file_header MatFile5Writer.write_file_headerNshhr<( >rwwiH--1\\^,<>MIZZb,0-/YYv->@M s{{}-r-c6UcURR5S:HnU(aUR5 [U5UlUR 5GH=up4USS:XaMX0R ;nUR(a[5nX`RlURRXCRS5U5 [R"UR55n[R"S[ 5n["US'[%U5US'URR'UR)55 URR'U5 GMURRXCRS5U5 GM@ g)aWrite variables in `mdict` to stream Parameters ---------- mdict : mapping mapping with method ``items`` returns name, contents pairs where ``name`` which will appear in the matlab workspace in file load, and ``contents`` is something writeable to a matlab file, such as a NumPy array. write_header : {None, True, False}, optional If True, then write the matlab file header before writing the variables. If None (the default) then write the file header if we are at position 0 in the stream. By setting False here, and setting the stream position to the end of the file, you can append variables to a matlab file Nrrrrr{r|)rrwrrrrr~r}rr%rTzlibcompressgetvaluer&emptyrrrrr) rHrrrvarrr~out_strr s r+ put_variablesMatFile5Writer.put_variablesYs1$  ++002a7L   " " $(.IDAw#~ 0 00I"" 28##/##--c;;x3H)T--(9:hhr<0 ,H $'LL!  &&s{{}5  &&w/##--c;;x3H)T'r-)rr}rr~rrr)FFNFrowr) rrrrrrrBrrrrr-r+r{r{.s1( %!&!"' ##8 .&Ur-r{)?rrPrrrCriorrnumpyr& scipy.sparser,_byteordercodesrr_miobaserrr r r r r rr _mio5_utilsr _mio5_paramsrrrrrrrrrrrrrrrr r!r"_streamsr#r,r1r0r<r>rrrrrr rrr{rr-r+rs EN 6444 $CCCCC &-    a]aHAHHH5r{#H-m< {#H-j9  $X.? +&x0?kk\ QUQUr-