• Welcome to Smashboards, the world's largest Super Smash Brothers community! Over 250,000 Smash Bros. fans from around the world have come to discuss these great games in over 19 million posts!

    You are currently viewing our boards as a visitor. Click here to sign up right now and start on your path in the Smash community!

Melee dat format...

revel8n

Smash Cadet
Joined
Dec 1, 2009
Messages
51
Figured i would break this out of the Melee Hacks main thread, to allow for better access to the available information, and hopefully gather more hex editing and reverse engineering enthusiasts to help sort out the various data locations and containing structures.

---------------------------

Pokémon Stadium without transformation : http://www.megaupload.com/?d=7UMQJ727

@Milun : How can I find the collision data ? Randomly, trying everywhere ?..
I spend hours to find the position data of the part I want (to many position data O_o) and I think finding collision data is something like impossible to me...
hey revel...
I still don't exactly understand how root nodes work :/

but I've thought maybe root nodes are offset indexes

such as, if a root node was set at '80', then it initiates at offset[128] and so on,
until it hits another node index such as 'F0',
then node '80' is terminated and node 'F0' takes it's place (reading from offset[240] and so on)

also, you never replied to my Q about the 16bit hex before the root node offsets...
it's only either '00 00' for normal costumes, or '00 01' for colord costumes...
what is that, and what does it do??
Well collision data is always a bit of a scroll above the magnifier. You can tell you've found it when you see a relocation table, a.k.a the thing I put a green X over on this page.

Hope that helps.
i guess i'll explain it again here and see if any others can pick it up and understand it further.

dat, usd, and certain other formats derived from the same format are all structured in a similar way. The file layout goes:

File Header
Data Block
Relocation Table
Root Nodes (2)
String Table

The file starts with a 0x20 (32) byte header which gives information on how to access the general archive structure.

Code:
struct DAT_HEADER
{
// 0x00
uint32 fileSize0x00                <format = hex>;
uint32 dataBlockSize0x04            <format = hex>; // size of main data block
uint32 relocationTableCount0x08;
uint32 rootCount0x0C;
// 0x10
uint32 rootCount0x10;
uint32 unknown0x14;                                // '001B' in main Pl*.dat files
uint32 unknown0x18;
uint32 unknown0x1C;
// 0x20
};


The start of the header starts with the 32-bit (4-byte) size of the file upon creation, which can be used to verify the file is the correct length.

The next value is 32-bit as well and gives the length of the data section that follows directly after the header. Given that main data section starts immediately after the header, the base offset within the file is 0x20 bytes from the beginning of the file. All file offsets found later in the file start from the beginning of this data section, so taking the entire file into account all offsets will be 0x20 bytes less than you would expect (as you have to add the base data offset). If this is too confusing, copying all data from directly after the header, for the size given in the header, and placing it in a separate file will allow all the offsets in the file to be treated without modification.

Now that you have the header and data sections sorted out, they are not much use without knowing where various information begins within the data.

Starting directly after the relocation table is a list of file offsets. The number of which is given in the header by relocationTableCount. Each file offset is 32-bit and is relative to the beginning of the data section. The purpose of the relocation table is not to give access to specific data structures, but to show the location of every other file offset located within the data section. This is done because when the file is loaded into memory by the game the memory location assigned by the main operating system will not be based on a zero offset like loading the file simply in a hex editor. As such all file offset within the data will still be relative to the beginning of the data and not absolute addresses in memory. Using the relocation table all file offsets within the file can easily be modified to be absolute instead of relative. If you are not planning on doing any real coding with this format, or have no issue using the other file offsets in a relative manner, then this information can be ignored entirely in most cases. The main benefit this data provides for those not using it to convert absolute memory addresses, is that it gives the location of every other file offset within the data. Given knowledge of how structures work in code, it is highly unlikely for an offset/pointer to jump into the middle of a structure. Given this very strong assumption, it can be assumed that any file offset is going to point to the location of the beginning of another data structure. This can be very useful in determining the start and end of a particular data structure, as they should not overlap. So if you start at the beginning of a structure you do not know yet and happen to cross an address that is accessed through another file offset, you have a strong indicator of where the structure should stop. If there is a count somewhere you can divide the size of the area by it and get individual structure lengths and so on.

After the relocation table there are two lists of 8-byte structures i have simply termed root nodes. These structures contain 2 values, a file offset followed by a string table offset.

Code:
struct ROOT_NODE
{
uint32 rootOffset0x00        <format = hex>;
uint32 stringTableOffset0x04 <format = hex>; // offset to name string
};
The number of the structures in the two lists is given by the counts, rootCount0x0C and rootCount0x10, the latter often being 0 in some files.

The rootOffset is relative to the beginning of the data section, while the stringTableOffset is relative to the beginning of the string table that follows directly after the root node lists.

As stated the string table is located directly after the root node lists. The size is arbitrary and not specified in the file. You could subtract the size of the other sections combined from the total file size if desired, as the files are terminated by the string table information. The number of strings in the table is given by rootCount0x0C + rootCount0x10, though the main way of accessing data from this area is through the offsets given in the root nodes.

Some equations for the various file relative start offsets for clarification:
Code:
local int64 dataOffset  = 0x20;
local int64 relocOffset = dataOffset + fileHeader.dataBlockSize0x04;
local int64 rootOffset0 = relocOffset + (fileHeader.relocationTableCount0x08 * 4);
local int64 rootOffset1 = rootOffset0 + (fileHeader.rootCount0x0C * 8);
local int64 tableOffset = rootOffset1 + (fileHeader.rootCount0x10 * 8);
Now, on to the interpretation of the data the root nodes point to.

So far i have found no other identifying information for the starting structures other than a naming scheme followed by the names of the root nodes given by their string table offset.

For instance:
Pl*.dat - The main player data files, usually contain a root node whose name begins with "ftData".
Pl*Aj.dat - Player animation related files (Which actually are containers for multiple sub dat files reference through the main Pl*.dat file), usually contain a root node name that contains "figatree".
Ty*.dat, Pl*Nr.dat - And other player color related files, can contain multiple nodes usually ending with "_joint" with other possible sub-strings being "matanim", "shapeanim" or assumed MATerial and SHAPE ANIMation respectively.
Gr*.dat - Stage related files can contain a large number of nodes with sub names like "_image" (which you may note contain a format identifier as well), "_tlut", "_tlut_desc", "coll_data", "map_head", and a number of others.

As an example:
Code:
if (Strstr(nodeName, "_joint") != -1 &&
Strstr(nodeName, "shapeanim") == -1 &&
Strstr(nodeName, "matanim") == -1)
{
// NOTE: Used for Ty*.dat and Pl*Nr.dat files
struct
{
DumpJObj(rootOffset0x00);
} jobjData;
}
else if (Strstr(nodeName, "ftData") != -1)
{
// NOTE: Used for Pl*.dat files
DumpFighter(rootOffset0x00);
}
else if (Strstr(nodeName, "map_head") != -1)
{
// NOTE: Used for Gr*.dat files
struct MAP_HEAD stageData;
}
else if (Strstr(nodeName, "figatree") != -1)
{
// NOTE: Used for Pl*Aj.dat files
struct FIGATREE_DATA animationData;
}
else if (Strstr(nodeName, "coll_data") != -1)
{
// NOTE: Used for Gr*.dat files
struct COLL_DATA collisionData;
}
i have only investigated a number of these sections so far, mainly those related to gaining access to joint information which contains the hierarchy and also branches off into various data structures related to materials, textures, palettes, geometry and mesh information. All of which can give you access to data like the location, width, height, and format information of image data. The format and number of colors found in palette data. The vertex attributes, joint weighting, and other mesh parameters. And so on...

Code:
struct JOBJ_DATA
{
// 0x00
uint32 unknown0x00      <format = hex>;
uint32 flags            <format = hex>;
uint32 childOffset      <format = hex>; // child jobj structure
uint32 nextOffset        <format = hex>; // next jobj structure
// 0x10
uint32 dobjOffset        <format = hex>; // dobj structure - object information?
float3 rotation;                        // rotation
float3 scale;                            // scale
float3 translation;                      // translation
uint32 transformOffset  <format = hex>; // inverse transform
uint32 unknown0x3C;
// 0x40
};


This is the main Joint object structure, this can usually be found on root nodes that end with "_joint" that do not contain "matanim" or "shapeanim" (usually with "TopN" or "Share" instead). Following this hierarchy of structures you can obtain all the information i am currently using to produce the screenshots i have posted.

As for the collision data mentioned, i had not looked into it until recently and have not deciphered any more than has already been posted. The data is found on the "coll_data" root node and the main structure follows this layout:
Code:
struct COLL_DATA
{
uint32 vertexOffset <format = hex>;
uint32 vertexCount;
uint32 indexOffset  <format = hex>;
uint32 indexCount;
struct
{
uint16 indexStart;
uint16 indexCount;
} unknownData0x10[5];
uint32 unknownOffset0x24 <format = hex>;
uint32 unknownCount0x28;
};
The offset i have termed vertexOffset points to the data outline in Milun's information (i have only given it that name, not actually investigated any further). The data is 2 float values per entry which i am guess as outlined are 2D position values.

The indexOffset points to the data crossed out in Milun's explanations, the other offset points to somewhat similar information, but i have not gone deeper than this yet.

Well, overall that is the main information you need to at least start accessing any data you want within the files. i have posted more information between here and the emutalk thread, as well as a template file that outlines this information. i will see if i can accompany this post with pictures and further explanations. Feel free to ask for any other specifics i may have already deciphered. You can get a good idea of what that is from what i have posted in my screenshots. As always any help deciphering and/or placing more know information is greatly appreciated.

Hope it helps.
 
Last edited:

revel8n

Smash Cadet
Joined
Dec 1, 2009
Messages
51
Joint Data - Accessing Material and Texture information

The joint structure outlined above is the main starting point of the data found within Ty*.dat and Pl*Nr.dat files, and can also be found within Stage, Item, Interface, Fighter, and other structures. These structures are the main containers for hierarchy information, as well as pointers to material and geometry information.

For reference:
Code:
struct JOBJ_DATA
{
// 0x00
uint32 unknown0x00      <format = hex>;
uint32 flags            <format = hex>;
uint32 childOffset      <format = hex>; // child jobj structure?
uint32 nextOffset        <format = hex>; // next jobj structure?
// 0x10
uint32 dobjOffset        <format = hex>; // dobj structure - object information?
float3 rotation;                        // rotation
float3 scale;                            // scale
float3 translation;                      // translation
uint32 transformOffset  <format = hex>; // inverse transform
uint32 unknown0x3C;
};


The hierarchy is controlled through following the next and child joint offsets. Using the transform information, combining child transforms with thier parent, you can calculate the final absolute transform of each joint location. These are used to position various elements within the data when it come to meshes and other geometry, and will be mentioned again later.

The flags here are currently still being investigated (help appreciated as always, heh...).

The other offset here leads to a linked list of structures containing further offsets to material and geometry information:

Code:
struct DOBJ_DATA
{
uint32 unknown0x00;
uint32 nextOffset <format = hex>; // next dobj structure
uint32 mobjOffset <format = hex>; // mobj structure - material information?
uint32 pobjOffset <format = hex>; // pobj structure - mesh information?
};


If valid you can follow the nextOffset value to obtain information on all material and mesh data associated with the particular joint it is linked to.

The mobjOffset structure:

Code:
struct MOBJ_DATA
{
// 0x00
uint32 unknown0x00;
uint32 unknownFlags0x04  <format = hex>;
uint32 tobjOffset        <format = hex>; // tobj structure
uint32 materialOffset    <format = hex>; // material colors?
// 0x10
uint32 unknown0x10;
uint32 unknown0x14;
};


contains offsets to the texture and material color information. Again there is another set of flags whose purpose is no full deciphered. Untextured materials may not contain a valid texture object offset.

The material color structure is as follows:

Code:
struct MATERIAL_COLORS
{
// 0x00
uint32 unknownColor0x00 <format = hex>; // diffuse?
uint32 unknownColor0x04 <format = hex>; // ambient?
uint32 unknownColor0x08 <format = hex>; // specular?
float  unknown0x0C;
// 0x10
float  unknown0x10;
};


i am still playing with the values as they were not necessary for creating my mesh viewer in its current state, but will be useful to those looking to further modify the appearance of models.

The texture information structure:

Code:
struct TOBJ_DATA
{
uint32 unknown0x00[19];
uint32 imageOffset      <format = hex>; // image header information
uint32 paletteOffset    <format = hex>; // palette header information
uint32 unknown0x54;
uint32 unknownOffset0x58 <format = hex>;
];


Contains information relating to the texture environment parameters used during rendering and offsets to the image and palette data used for this texture. The 19 values at the beginning of the structure contain information that affects rendering in game, but i have not deciphered them yet.

Other than that the image and palette/tlut information offsets are the most important data here. For images that are not indexed (RGBA, CMPR, etc.) the palette/tlut description structure may not be used.

The relevant structures are as follows:

Code:
struct IMAGE_HEADER
{
uint32 imageOffset <format = hex>; // image data
uint16 width;
uint16 height;
uint32 imageFormat;
};

struct PALETTE_HEADER
{
uint32 paletteOffset <format = hex>; // palette data
uint32 paletteFormat;
uint32 unknown0x08;
uint16 colorCount;
uint16 unknown0x0E;
};
The data offsets in each structure lead to their respective data locations and are often shared amongst multiple textures. From this information you can gather everything you need to determine the width, height, and format of the image data, as well as the format and color count of the palette/tlut data.

Palettes do no have to use the full number of colors that would otherwise be expected by the image data format. For instance, an 8-bit indexed image as a mximum possible color count of 256, but only 136 or 221 of those colors may actually be present. As such the palette data may not occupy the full range of space you would otherwise think.

The image formats follow traditional gamecube layouts, and have been outlines in other threads, but here they are for further reference:
Code:
Image formats:

case 0: //i4
case 1: //i8
case 2: //i4a4
case 3: //i8a8
case 4: //r5g6b5
case 5: //rgb5a3
case 6: //r8g8b8a8
case 8: //index4
case 9: //index8
case 0xa: //index14x2
case 0xe: //s3tc1
Indexed formats also make use of palette/tlut information and the data can also appear in multiple formats:

Code:
Palette formats:

case 0: //ia8
case 1: //r5g6b5
case 2: //rgb5a3

With this information you can obtain the location of each texture within the file once you find the initial joint information structure. This is far easier with Ty*.dat (Toy) and Pl*Nr.dat (Player color) files as the "_joint" root nodes that are not related to "matanim" or "shapeanim" data start with a joint structure and the hierarchy can be followed to the material and texture information.

As always, any help testing and deciphering the unknown portions is greatly appreciated. Anyone with access to more information on the internals of the sysdolphin api would definitely be of great help.

If need i can clarify more things should it be needed.

i plan on continuing with the mesh data structures in another post.

Hope this helps.
 
Last edited:

revel8n

Smash Cadet
Joined
Dec 1, 2009
Messages
51
Joint Data - Accessing Geometry, Mesh, and Vertex Attributes

Ok, now moving on to interpreting the mesh information present within the joint data hierarchy.

As mentioned in the previous post, the main data structure found off of the joint structures contains offsets to material and geometry information.

The material structures were covered previously, the other offset leads to this structure:
Code:
struct POBJ_DATA
{
	// 0x00
	uint32 unknown0x00;
	uint32 nextOffset        <format = hex>;
	uint32 vertexAttrArray   <format = hex>; // vertex attribute list
	uint16 unknownFlags0x0C  <format = hex>;
	uint16 displayListSize0x0E;              // number of 0x20 (32) byte blocks occupied by display list data
	// 0x10
	uint32 displayListOffset <format = hex>; // display list
	uint32 weightListOffset  <format = hex>; // joint weight info
};
As evident by the nextOffset value this structure can actually be a list of multiple structures representing a number of meshes to be draw with the given material also referenced by the DOBJ structure. This structure contains offsets to the vertex attributes, display list, and joint weight list data, and is mostly everything you need to properly interpret data pertaining to processing vertices, normals, and texture coordinate information correctly.

The vertex attributes are probably the most important aspect of this data. The parameters specified control everything from the format, and thus data size of each vertex, normal, and texture coordinate and how those values are scaled to the presence and size of each index value that appears in the display list information.

As such it is probably best to start with that structure and the information it contains:
Code:
// vertex declaration and attribute information
// attr, type, cnt, data_type, flags?, file_offset
struct ATTR_DATA
{
	// 0x00
	GXAttr     vtxAttr;     // attr
	GXAttrType vtxAttrType; // index_type
	GXCompCnt  compCnt;     // cnt
	GXCompType compType;    // data_type	
	// 0x10
	uint8      scale;
	uint8      unknown0x11;
	uint16     vtxStride    <format = hex>;
	uint32     dataOffset   <format = hex>;
};
The vertexAttrArray offset points to a list of multiple vertex attributes. The count is not given. The data in this structure follows information and types designated by the gamecube/wii architection and API design. The list is terminated by the vtxAttr value being setting to a specific value signaling the termination of this list. If you have access to the relevant sdk documentation it would be highly recommended to read the sections related to vertex attributes and other specifications.

The vtxAttr value specifies the type of data this attribute entry is referring to, and can indicate a number of native types used to pass data between software and the hardware interfaces.

The possible values of this are as follows:
Code:
enum <uint32> GXAttr
{
    GX_VA_PNMTXIDX = 0,    // position/normal matrix index
    GX_VA_TEX0MTXIDX,      // texture 0 matrix index
    GX_VA_TEX1MTXIDX,      // texture 1 matrix index
    GX_VA_TEX2MTXIDX,      // texture 2 matrix index
    GX_VA_TEX3MTXIDX,      // texture 3 matrix index
    GX_VA_TEX4MTXIDX,      // texture 4 matrix index
    GX_VA_TEX5MTXIDX,      // texture 5 matrix index
    GX_VA_TEX6MTXIDX,      // texture 6 matrix index
    GX_VA_TEX7MTXIDX,      // texture 7 matrix index
    GX_VA_POS      = 9,    // position
    GX_VA_NRM,             // normal
    GX_VA_CLR0,            // color 0
    GX_VA_CLR1,            // color 1
    GX_VA_TEX0,            // input texture coordinate 0
    GX_VA_TEX1,            // input texture coordinate 1
    GX_VA_TEX2,            // input texture coordinate 2
    GX_VA_TEX3,            // input texture coordinate 3
    GX_VA_TEX4,            // input texture coordinate 4
    GX_VA_TEX5,            // input texture coordinate 5
    GX_VA_TEX6,            // input texture coordinate 6
    GX_VA_TEX7,            // input texture coordinate 7

    GX_POS_MTX_ARRAY,      // position matrix array pointer
    GX_NRM_MTX_ARRAY,      // normal matrix array pointer
    GX_TEX_MTX_ARRAY,      // texture matrix array pointer
    GX_LIGHT_ARRAY,        // light parameter array pointer
    GX_VA_NBT,             // normal, bi-normal, tangent 
    GX_VA_MAX_ATTR,        // maximum number of vertex attributes

    GX_VA_NULL     = 0xff  // NULL attribute (to mark end of lists)
};
As you can see, any of the data you would think necessary to represent a mesh can be specified and indexed in some way. Although GX_VA_TEX0MTXIDX, GX_VA_CLR0, and others are possible as well, the most commonly used values within Melee data are GX_VA_PNMTXIDX, GX_VA_POS, GX_VA_NRM, and GX_VA_TEX0. Which is to be expected as joint matrices, vertex positions/normals, and texture coordinates respectively are probably the most basic of elements needed to display a texture mesh, whether static or animated.

When the value of vtxAttr == GX_VA_NULL (0xFF), the end of the vertex attribute array has been reached.


The vtxAttrType values is actually associated with how the value is indexed, and can thus also determine the size of the index value within the display list data.
Code:
enum <uint32> GXAttrType
{
	GX_NONE    = 0,
	GX_DIRECT,
	GX_INDEX8,
	GX_INDEX16
};
INDEX8 and INDEX16 are pretty self explanatory in that the index value used within the display list will be either 8- or 16- bit respectively.

The interpretation of the attribute type when DIRECT is specified is determined by the attribute that is being referenced. Only certain attributes can be specified directly, the main ones being Matrix indices and Color values. For clarification of what this implicates, for instance, if a color value (CLR0, etc.) was specified as a DIRECT attribute, it would not have a separate data section like vertices and normals, the data itself would be included explicitly in the display list data along with the other index values. More on this in a bit, though again the sdk documentation is helpful here.

The compCnt and compType, component count and component type, values are also dependent on the attribute being referenced.
The component count basically determines the dimension of the data - whether positions are XY (2D) or XYZ (3D), whether colors are RGB or RGBA, whether texture coordinates are S (1D) or ST (2D), whether normals also contain tangents or not, etc.
The component type specifies the data type used to represent the data - signed or unsigned integers, floats for values or 16- or 32-bit for colors, etc.

Component Count:
Code:
enum <uint32> GXCompCnt
{
	GX_DEFAULT    = 0,
};

Position Counts:
enum <uint32> GXPosCompCnt
{
	GX_POS_XY    = 0,
	GX_POS_XYZ   = 1,
};

Normal Counts:
enum <uint32> GXNrmCompCnt
{
	GX_NRM_XYZ   = 0,
	GX_NRM_NBT   = 1, // one index per NBT
	GX_NRM_NBT3  = 2, // one index per each of N/B/T
};

Color Counts:
enum <uint32> GXClrCompCnt
{
	GX_CLR_RGB   = 0,
	GX_CLR_RGBA  = 1,
};

Texture Coordinate Counts:
enum <uint32> GXTexCompCnt
{
	GX_TEX_S     = 0,
	GX_TEX_ST    = 1,
};
Component Type:
Code:
Value Types:
enum <uint32> GXCompType
{
    GX_U8  = 0, // 0 - 255
    GX_S8  = 1, // -127 - +127
    GX_U16 = 2, // 0 - 65535
    GX_S16 = 3, // -32767 - +32767
    GX_F32 = 4, // floating point
};

Color Types:
enum <uint32> GXClrCompType
{
	GX_RGB565 = 0,
	GX_RGB8   = 1,
	GX_RGBX8  = 2,
	GX_RGBA4  = 3,
	GX_RGBA6  = 4,
	GX_RGBA8  = 5,
};
The vtxStride value is the data length of a single complete attribute value, the distance between the start of one value and the next.

As an example if you had an attribute describing position data with the following parameters:
Code:
GX_VA_POS, GX_INDEX16, GX_POS_XYZ, GX_F32
The attribute described would contain vertex position data, indexed by a 16-bit value, where the data contains 3 values per position each being single precision floating point, as we would expect for a floating point 3d vector. With this in mind, each position contain 3 values, one each for X, Y, and Z respectively, with a 32-bit (4-byte) float value per component - each vertex position would occupy 12-bytes of space. If the same attribute specified each component as being a signed byte (S8) then each vertex would only take up 3-bytes total, and so on.
The size computed above should equal the vtxStride value in most, if not all cases. With this you can easily calculate the amount of data to move when indexing into the data for a particular attribute.

Now that you know the format of the attribute, you can easily determine how to convert the data into a form necessary to use it for rendering or other processing. but there is one more things that can affect the interpretation of the data and its actual range of values, the scale.

For conversion other than F32, the interpretation of the values obtained are not immediately known. You can take the value literally and use it as is, you can normalize the value by dividing by its maximum to get a range between 0 and 1 or some other arbitrary range, etc.

The scale is a power of 2 exponent that give you the divisor for the values in order to convert them into the range of values intended.

As an example, given a signed 8-bit value, whose range of literal values would be about -127 to +127, in order to normalize the values, you might divide by 127.0 to get the result in a range of -1.0 to +1.0. This may or may not be the intended result depending on the attribute data in question. Instead you would compute 2^scale and use that value as the divisor for all component values of this attribute.

So a scale of 6 would be come 2^6 or 64, thus all values should be divided by 64 (or multiplied by (1.0/64.0)) to get the correct range of values. This is a VERY important step to take into account as this value may vary greatly from what you might expect to be the default.

i do not worry about a scale value of 0, which is common for float attributes, because 2^0 is 1 and thus can be handled in the same way without extra consideration. For attributes where this value does not make sense, like colors, this can usually be disregarded.

Well, now knowing how to interpret the data is pretty useless if you cannot find it, so the dataOffset should come in very handy for this purpose. There has been a lot of assumption made with regard to vertices, normals and texture coordinates starting in a specific location and appearing in a consistent order. Most of the time these assumptions (0x20 for vertices, followed by normals, then texture coordinates) hold true in most cases, but is not guaranteed to be the case, especially if a given mesh does not even contain some of these attributes, has others, etc. It is also good to have this information for those looking to be able to increase the size of data or better yet move data to other locations entirely (which may be necessary if you desire to change the format of data, etc.).


With this so far you can determine the attributes used by a mesh, the format of those attributes, how to interpret the values of the attributes, as well as their location, but usually all data of a similar type and format are simply lumped together in the same data section, even if some of the data pertains to multiple meshes. As such you also need to know how to index the data of each attribute.

This leads us to the information pointed to by the displayListOffset value.

This value points to display list information formatted in a standard way the gamecube/wii api expects to find it. As has been notice with many gamecube games, data of this type is quite common as since it is the way the application natively expects it, it is also the fastest way to specify and send it to the hardware, much like the attribute data.

The size of the display list data is given by the displayListSize value * 32. Much like other data that is intended to be sent to the system hardware the data is often aligned to start on a 32-byte boundary, so the space allocated for the data is a multiple of 32 bytes.

The basic display list format starts with this information:
Code:
struct DISPLAY_LIST
{
	uint8  primitiveFlags <format = hex>;
	uint16 indexCount;
};
The primitiveFlags value specifies the type of rendering primitive as well as the vertex stream to use. The primitive type is the important bit of this value and occupies the upper bits of the 8-bit value and should be masked with 0xF8 to ensure the vertex stream index is removed.

Once that is done the primitive type can take the following values:
Code:
Primitive Types:

case 0xB8: // (GL_POINTS)
case 0xA8: // (GL_LINES)
case 0xB0: // (GL_LINE_STRIP)
case 0x90: // (GL_TRIANGLES)
case 0x98: // (GL_TRIANGLE_STRIP)
case 0xA0: // (GL_TRIANGLE_FAN)
case 0x80: // (GL_QUADS)
These are simple rendering primitives supported by most rendering API's so there is plenty of information on the subject. Most recent API's being deigned similarly to OpenGL as you can see i have related them to their respective formats for that API.

The display list information is not just one continuous stream of a single primitive's data. Multiple strips can occur within a single display list section. and the primitive type can change between them as necessary.

The data that follows this header contains the index information, and any values that are specified DIRECT-ly based on the vertex attribute data. The number of index values is given in the display list header, and each set of indices contains and entry for each attribute in the vertex attribute list. Thus the size of each complete index entry is the sum of the index data specified for each attribute. INDEX8 is and 8-bit (1-byte) index and INDEX16 is 16-bits (2-Bytes). DIRECT is handled a bit differently. For non color values that are specified directly, the value is 8-bits. For color values the element is the size of the specified color type (RGB8 is 24-bit (3-bytes), RGBX8/RGBA is 32-bit (4-bytes), RGB565 is 16-bit (2-bytes), and so on...).

The indices and DIRECT values appear in the order they were specified in the vertex attribute data.

Example 1 (PlFxNr.dat):
Code:
Vertex Attributes (0xD37C):

00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 04
00 00 00 00 00 00 00 00 00 00 00 09 00 00 00 03
00 00 00 01 00 00 00 03 0A 00 00 06 00 00 00 00
00 00 00 0A 00 00 00 03 00 00 00 00 00 00 00 01
06 00 00 03 00 00 41 E0 00 00 00 0D 00 00 00 03
00 00 00 01 00 00 00 03 0D 00 00 04 00 00 65 A0
00 00 00 FF 00 00 00 02 00 00 00 00 00 00 00 04
00 00 00 00 00 00 00 00
Code:
Display List (0xD460):

90 00 09 15 07 C1 0A DA 09 93 18 07 D8 0A F1 09
94 18 07 BF 0A D8 09 AB 0C 07 CF 0A EA 09 BE 0F
07 A7 0A 50 08 49 0F 07 B7 0A CF 09 A8 15 07 A4
09 E3 08 3D 15 07 D4 09 DF 09 C3 15 07 A2 09 E0
08 3B 80 00 0C 0C 07 CC 0A E7 09 BB 0F 07 CB 0A
E6 06 A9 0F 07 BA 0A D3 09 A7 0F 07 BB 0A D4 09
A8 0F 07 B8 0A D0 09 A5 0F 07 D0 0A EB 06 AA 0C
07 CF 0A EA 09 B7 0F 07 B7 0A CF 09 A4 15 07 9D
09 D9 08 23 15 07 9A 09 D6 08 20 15 07 9B 09 D7
08 21 15 07 9C 09 D8 08 22 .. .. .. .. .. .. ..
Above you can see the vertex attribute data and a partial excerpt of the display list data for a mesh.

Looking at the attribute data, you should be able to parse the information and determine the following:
Code:
GX_VA_PNMTXIDX, GX_DIRECT,  GX_DEFAULT, GX_F32,  0,  0,  0,  0
GX_VA_POS,      GX_INDEX16, GX_POS_XYZ, GX_S16, 10,  0,  6,  0
GX_VA_NRM,      GX_INDEX16, GX_NRM_XYZ, GX_S8,   6,  0,  3,  41E0
GX_VA_TEX0,     GX_INDEX16, GX_TEX_ST,  GX_S16, 13,  0,  4,  65A0
GX_VA_NULL,     GX_INDEX8,  GX_DEFAULT, GX_F32,  0,  0,  0,  0
Here you have a vertex with 4 attributes - a matrix index, position, normal, and a single set of texture coordinates. The matrix index is specified directly and will be present within the index data itself. The GX_VA_NULL entry terminates the list and can otherwise be ignored.

The vertex data contains X, Y, and Z, each 2-bytes (S16) for a total of 6-bytes per vertex with a scale value of 2^10...
The normal data contains X, Y, and Z, each 1-byte (S8) for a total of 3-bytes per normal with a scale value of 2^6...
The texture coordinate data contains U and V, each 2-bytes (S16) for a total of 4-bytes per texture coordinate with a scale value of 2^13...

The DIRECT entry is 8-bit (1-byte) and each index is 16-bits (2-bytes), for a total of 7-bytes per index set. Thus looking a the display list data we see a primitive flag of 0x90 specifying an indexed triangle list, and an index count of 0x0009. At 7-bytes each there are 0x3F (63) byte of data contained in this listing.
This would make the first index set be:
Code:
[15] [07C1] [0ADA] [0993]
After the 9 index set you can see that another list starts specifying a primitive type of 0x80 (QUADS) with an index count of 0x0C (12)...

The display list data continues in this fashion until you reach the end of the section give by the display list size, or reach an invalid primitive flag specification (usually 0).

Hope that helps, just ask if anything needs to be clarified further.
 

revel8n

Smash Cadet
Joined
Dec 1, 2009
Messages
51
Hopefully i have made a slight bit of progress. Going through the texture information i believe i have found a bit more information.

Code:
struct TOBJ_DATA
{
    // 0x00
    uint32 unknown0x00[13];
    uint32 wrap_s;
    uint32 wrap_t;
    uint8  widthScale;
    uint8  heightScale;
    uint16 unknown0x3E;
    // 0x40
    uint32 unknown0x40[3];
    uint32 imageOffset       <format = hex>; // image header information
    // 0x50
    uint32 paletteOffset     <format = hex>; // palette header information
    uint32 unknown0x54;
    uint32 unknownOffset0x58 <format = hex>;
    // 0x5C
};
The wrap modes seem to correspond to the enumerations given within the gamecube/wii documentation.

Code:
0 - GX_CLAMP
1 - GX_REPEAT
2 - GX_MIRROR
The scales simply seem to further modify the texture coordinates, and so far i do not know the acceptable range, as so far i have 1, 2, and 10.

So far is seems to pan out as a lot of the flipped and incorrectly tiled images are now showing correctly:


Once proper blending is found and taken into account, it should hopefully get things even closer to how it is in game. i especially want to get to that point for stages.

Have some more testing to do regarding joint data as well, hopefully to make a further attempt at deciphering the information in the animation data.
 
Last edited:

revel8n

Smash Cadet
Joined
Dec 1, 2009
Messages
51
Any information on the files would be useful. Once i can get the flags and other data sorted out i will be that much closer to determining the rest of the information. i have only done marginal work with the figatree files and still have note determined how they related to the hierarchy found within the main player/color files. The structure i have so far can allow me to access all the player attribute data, i am just working on determining how it all fits together.

If i i could get to a point where i could also animate the characters or even track what effects the various moveset information i may be able to relate more things.

So any information could be very useful.
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
finally

hopefully I'll make progress now...

I'll be referencing this thread on my SBF update btw ;)

EDIT:
hey revel <:|

just to make things quicker,
may I send you my template,
and can you get it to work like yours??

you are way ahead on everything and I'm only just starting...

or at least can you explain your stuff a little different...
I can't piece together how you've worded it...

eg:

0x00(4): filesize
0x04(4): reloc tbl offset
0x08(4): num offsets
...

to find the relocation table:
reloc tbl offset + 32

read (4) for(num offsets > 0)
....jump to offset + 32
....num offsets - 1


^that's as far as I know really :/
what do I do next??

I didn't go into detail with it...
but you get the idea...

can you please explain it more like that :|

EDIT:
REVEL!

you didn't tell me there was a 2nd offset...
you've set me back dangit!

I kept asking and you kept giving me the wrong info

relocation offset + 32
data offset + 32 (from relocation offset)

I finally got somewhere now...
and it's been like what... 2 years and no progress...

can't believe I had to go to GodFed to get my answer...
geeze...

^glad I got that off my chest...

now, that aside...
I've figured out a little something about the relocation table:


wanna I'm going to look into it now that I know what to do...
you've provided me with enough needed info there...
 
Last edited:

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
^revel8n is a good dude, don't get mad at him, Tcll. Just keep moving forward with what you've got.
: )
I know he's a good dude...

but it's been years with no progress on my end...
that's what P's me...

I'm moving foreward right now btw
got alot of catching up to do on this -.-

he's definatly getting credit in my converter >:)

dude he helped me on my brute-force method, which was at least something.
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
hey revel
screw me giving you my template to work on for me >:)

I want to work on this myself and have wanted to...
it's just that I've been frustrated with no progress and was all like
HERE! you take it and do it

but no
I really want to be the one to do it, and I hate passing off my work... DX

anyways...
I want to ask,
can I get a copy of your recent template??

I just want it so I can build my own off of it...


I wish I could use 010E...
but noooooo they had to put a stupid trial time on it... DX<
leaving me to stick with stupid hexedit

btw GF...
abt hexedit...
what version are you using??

I'm using an old version as I know it's no 30 day trial
but you have a newer version :/
does it have the trial on it??

EDIT:
KK
so the relocation table takes me to a bunch of what looks like texture headers...
that is until offset 130 (offsets read by 5)

offset 130 looks like an oblect header...
but how do you tell the difference between the data so it doesn't read the polygon data as texture data??

is there an offset length to read by or something??

I feel like a noob -.-
man I hate being set back...

EDIT2:
here's the data at offset 130:


what is that data in blue??

this is PlPkNr.dat btw
 
Last edited:

revel8n

Smash Cadet
Joined
Dec 1, 2009
Messages
51
EDIT:
REVEL!

you didn't tell me there was a 2nd offset...
you've set me back dangit!

I kept asking and you kept giving me the wrong info

relocation offset + 32
data offset + 32 (from relocation offset)

I finally got somewhere now...
and it's been like what... 2 years and no progress...

can't believe I had to go to GodFed to get my answer...
geeze...

^glad I got that off my chest...

now, that aside...
I've figured out a little something about the relocation table:
http://lh3.ggpht.com/_IteXPmeC6ek/TRZzdz25WSI/AAAAAAAACP8/wLA9XFWUEls/discovery.jpg

wanna I'm going to look into it now that I know what to do...
you've provided me with enough needed info there...
http://www.emutalk.net/showpost.php?p=433176&postcount=250
http://www.emutalk.net/showpost.php?p=435593&postcount=445

Didn't know i misled you. i thought you seemed to understand things just fine from how you worded it yourself. If i can make things even easier to understand for others though, the better off we will all be.

http://www.emutalk.net/showpost.php?p=430453&postcount=161

i have stated here and in the original emutalk thread that all offsets, excepth for string table offsets, are relative to the beginning of the data section, and that the data section starts immediately after the file header, which is 32-bytes long.

i am trying to make some time to complete a few more things in this area that i hope will help. i will see if i can make some comments in my latest template, although everything in the template is explained above (outside of some data related to fighter, map, and other data structures i am still comparing).

And for reference, since Dantarion posted here, through searching i found his set of google code projects. http://code.google.com/u/dantarion/

OpenSA and meleetools being the most relevant here, as it finally shed light on the way moveset information worked. i hope to look further into the information within the Pl**Aj.dat files in hopes of making a further connection between how things are animated. i am hoping understanding more about how things are handled in brawl will help, but only time will tell.

Not sure how many other projects like this i have missed that might otherwise contain even more useful information.
 

GodFed

Smash Apprentice
Joined
Mar 25, 2009
Messages
160
Location
Hendrick House, Urbana, IL
I wish I could use 010E...

btw GF...
abt hexedit...
what version are you using??

EDIT:
KK
so the relocation table takes me to a bunch of what looks like texture headers...
that is until offset 130 (offsets read by 5)

offset 130 looks like an oblect header...
but how do you tell the difference between the data so it doesn't read the polygon data as texture data??

is there an offset length to read by or something??

I feel like a noob -.-
man I hate being set back...

EDIT2:
here's the data at offset 130:
http://lh3.ggpht.com/_IteXPmeC6ek/TRaw1p0eQjI/AAAAAAAACQQ/eQ0f-WxrIFk/Q.jpg

what is that data in blue??

this is PlPkNr.dat btw
Yeah, 010 has a trial period. You can work around that, though. As for hexedit, my version is 3.4, and it too has a trial expiration thing. It expired a while ago for me. All it does is nag me with a popup every so often, that's all.

I'm not sure what you mean when you talk about "offset 130." I checked it out myself, and it turns out that that specific location in the image (0x7494) is associated with the 130th relocation table value. I'll assume that's what you meant.
However, that value at 0x7494 is an offset pointing somewhere else in the file. It's not a data structure in itself. It just points to a data structure at offset 0x013198. I don't know what the data there is, exactly, but yeah.
 

revel8n

Smash Cadet
Joined
Dec 1, 2009
Messages
51
The data in blue is the beginning of a joint weight list. A list of structures containing a joint offset and float weight value, terminated with a zero offset and weight.

There is no way of telling this from the relocation table alone. You have to follow the root node structures.
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
The data in blue is the beginning of a joint weight list. A list of structures containing a joint offset and float weight value, terminated with a zero offset and weight.

There is no way of telling this from the relocation table alone. You have to follow the root node structures.
KK

and I did check the structures, but the offset they lead to is way at the bone data...

I think it may be the offset from 130 that goes beyond the root node offset...
but I'm not sure how to do the check to tell me that :/

how do you use the root nodes with the offsets...
(I think you may have stated that before)

Didn't know i misled you. i thought you seemed to understand things just fine from how you worded it yourself. If i can make things even easier to understand for others though, the better off we will all be.
that was back when I misunderstood the relocation table...
I thought it went directly to the data...

^thinking of it that way, you can see how everything you were previously telling me led me nowhere :p

and if you want to make things easier to understand,
try writing notes like this:

Code:
[COLOR=lime]REFT notes:[/COLOR]
 
[COLOR=lime]main header:[/COLOR]
[COLOR=lime]0x00(4): REFT[/COLOR]
[COLOR=lime]0x04(2): ??[/COLOR]
[COLOR=lime]0x06(2): ?? ('0007')[/COLOR]
[COLOR=lime]0x08(4): filesize[/COLOR]
[COLOR=lime]0x0C(2): ?? ('0010')[/COLOR]
[COLOR=lime]0x0E(2): internal file count[/COLOR]
 
[COLOR=lime]internal file header(0x10)[/COLOR]
[COLOR=lime]0x00(4): REFT[/COLOR]
[COLOR=lime]0x04(4): filesize - 8[/COLOR]
[COLOR=lime]0x08(4): header length - 8 (from 0x18)[/COLOR]
[COLOR=lime]0x0C(4): '00 00 00 00'[/COLOR]
 
[COLOR=lime]header P2:--[/COLOR]
[COLOR=lime]0x20(4): '00 00 00 00'[/COLOR]
[COLOR=lime]0x24(2): string length [ '000B' = 11 ][/COLOR]
[COLOR=lime]0x26(2): ?? ('0000')[/COLOR]
[COLOR=lime]0x28(?): string [ len('46696E50696B6163687500') = 11 ][/COLOR]
[COLOR=lime]0x##(?): -pad until header length- (following string)[/COLOR]
 
[COLOR=lime]string table:[/COLOR]
[COLOR=lime]0x00(4): block size[/COLOR]
[COLOR=lime]0x04(2): # of strings[/COLOR]
[COLOR=lime]0x00(4): string length[/COLOR]
 
[COLOR=lime]-after string-[/COLOR]
[COLOR=lime]0x00(4): data offset - (header length + 24)[/COLOR]
[COLOR=lime]0x00(4): data offset length[/COLOR]
and revel...
sry for snapping off at you :/
I'm usually not the type to do that...
it's just... well you understand -.-

EDIT:
oh and revel...

I hate it when you say 'as stated ealier' or 'in other threads' etc...
you're not giving the user the info they need :/

I just copy the info you stated and quote it
or create a link to it

it's better to have more posts referencing the info to give the other person a better undetstanding

btw, have you been looking into the mdl0 format at all??
you did mention before that you were interested in it...
I can provide you with alot of info,
and I have a template I've been working on to help people understand the format ;)

I've posted it quite a few times...
but it's not complete yet...

I'ma try to understand the polygons a bit more before I UD again...
I'll UD you on that one if you want it :)

the polygon format is alot harder to understand in the mdl0 format so it'll be a little before I can make a release...
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
Yeah, 010 has a trial period. You can work around that, though. As for hexedit, my version is 3.4, and it too has a trial expiration thing. It expired a while ago for me. All it does is nag me with a popup every so often, that's all.
wait...
work around it??

I tried a keygen on it, but that didn't work...
it returned an invalad key error after the 30 days... -.-

I use HexEdit version 3.0, and there's no trial period I've run into ;)

but I have a slight hunch 3.1 or 3.2 may not have trial periods either... >_>
 

GodFed

Smash Apprentice
Joined
Mar 25, 2009
Messages
160
Location
Hendrick House, Urbana, IL
Yep, you can just uninstall it and reinstall it to reset the trial period. Its real easy.

Hexedit is okay, but I like the template functionality on 010 better. Especially with the dat files, everythings easier to see.
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
Yep, you can just uninstall it and reinstall it to reset the trial period. Its real easy.

Hexedit is okay, but I like the template functionality on 010 better. Especially with the dat files, everythings easier to see.
same here with HexEdit...

but I believe I tried that with 010E before...
it didn't work for me...
what version do you have??
that may be why...
 

revel8n

Smash Cadet
Joined
Dec 1, 2009
Messages
51
KK

and I did check the structures, but the offset they lead to is way at the bone data...
joint = bone
So the joint weight list is a list of offsets to a joint/bone and a weight value to use for that bone. Without the geometry data structure that links to this list it is not much use on its own.


how do you use the root nodes with the offsets...
(I think you may have stated that before)
To use the root nodes you first have to find the string value at the string table offset given in the root node structure. This string tells you what data to expect at this root node. Once you know what data is supposed to be there, you can either use a known structure that matches the string designator, or you will need to compare like structures in multiple files to attempt to create a mapping of the data.


and if you want to make things easier to understand,
try writing notes like this:

Code:
[COLOR=lime]REFT notes:[/COLOR]
 
[COLOR=lime]main header:[/COLOR]
[COLOR=lime]0x00(4): REFT[/COLOR]
[COLOR=lime]0x04(2): ??[/COLOR]
[COLOR=lime]0x06(2): ?? ('0007')[/COLOR]
[COLOR=lime]0x08(4): filesize[/COLOR]
[COLOR=lime]0x0C(2): ?? ('0010')[/COLOR]
[COLOR=lime]0x0E(2): internal file count[/COLOR]
 
[COLOR=lime]internal file header(0x10)[/COLOR]
[COLOR=lime]0x00(4): REFT[/COLOR]
[COLOR=lime]0x04(4): filesize - 8[/COLOR]
[COLOR=lime]0x08(4): header length - 8 (from 0x18)[/COLOR]
[COLOR=lime]0x0C(4): '00 00 00 00'[/COLOR]
 
[COLOR=lime]header P2:--[/COLOR]
[COLOR=lime]0x20(4): '00 00 00 00'[/COLOR]
[COLOR=lime]0x24(2): string length [ '000B' = 11 ][/COLOR]
[COLOR=lime]0x26(2): ?? ('0000')[/COLOR]
[COLOR=lime]0x28(?): string [ len('46696E50696B6163687500') = 11 ][/COLOR]
[COLOR=lime]0x##(?): -pad until header length- (following string)[/COLOR]
 
[COLOR=lime]string table:[/COLOR]
[COLOR=lime]0x00(4): block size[/COLOR]
[COLOR=lime]0x04(2): # of strings[/COLOR]
[COLOR=lime]0x00(4): string length[/COLOR]
 
[COLOR=lime]-after string-[/COLOR]
[COLOR=lime]0x00(4): data offset - (header length + 24)[/COLOR]
[COLOR=lime]0x00(4): data offset length[/COLOR]
Hmm, well i guess i'll outline a few things and compare the way i specified my structure so that is may help you convert the ones i may not get to in the post.

uint32 = 32-bits (4-bytes)
uint16 = 16-bits (2-bytes)
uint8 = 8-bits (1-byte)
float = 32-bits (4-bytes)
float2 = 2 float values = 2 x 4-bytes = 8-bytes
float3 = 3 float values = 3 x 4-bytes = 12-bytes
float4 = 4 float values = 4 x 4-bytes = 16-bytes

And so on for similar types...

So starting with the header structure i outlined:
Code:
struct DAT_HEADER
{
	// 0x00
	uint32 fileSize0x00                 <format = hex>;
	uint32 dataBlockSize0x04            <format = hex>; // size of main data block
	uint32 relocationTableCount0x08;
	uint32 rootCount0x0C;
	// 0x10
	uint32 rootCount0x10;
	uint32 unknown0x14;                                 // '001B' in main Pl*.dat files
	uint32 unknown0x18;
	uint32 unknown0x1C;
	// 0x20
};
This would become:
Code:
main dat header:
0x00(4): file size
0x04(4): size of data section
0x08(4): number of relocation table entries
0x0C(4): root node count
0x10(4): reference node count
0x14(4): ?? ('001B')
0x18(4): ??
0x1C(4): ??
following the 0x20 (32-byte) header there is the data section. The size is given by the value at 0x04 in the header.

Directly following the data section at offset:
[0x20 (header size)] + [data section size]
is the list of relocation table offsets
Code:
relocation table:
0x00(?): list of offsets to every other offset value in the data section (length = 4 x number of relocation table entries)
Every location specified in the relocation table points to another offset value somewhere in the data section. Since the data section starts after the 0x20 (32-byte) header, all offsets in the data section must have 0x20 added to it. All the offsets pointed to by the relocation table is also relative to the data section so they must also have 0x20 added to them.

Following the relocation table at offset:
[0x20 (header size)] + [data section size] + [4 x number of relocation entries]
is the list of root and reference node structures.
Code:
root/reference node:
0x00(4): data offset (relative to data section, so add 0x20)
0x04(4): string table offset (relative to string table)
The number of these structures is given by the the 2 count values in the header for the root and reference nodes.

so the size of the root/reference node section is given by:
size of root sections = 8 * (root count + reference count)

Following the root/reference nodes at offset:
[0x20 (header size)] + [data section size] + [4 x number of relocation entries] + [8 * (root count + reference count)]
is the string table.

The number of strings is given by the number of root and reference nodes specified in the header.

There are no lengths for the strings, they are simply terminated by a 0 byte at the end. The string table offsets given in the root node structures specify the starting offsets of the names of the root nodes.

So for the offset value calculated for the beginning of the string table, it gets added to the string table offsets given in the root node structures.

These names are fairly specific and can be relied upon to match up with certain structures.

Names that begin with "ftData" represent fighter related header data.
Names that begin with "coll_data" specify collision data.
Names that contain "map_head" contain information on stage header data.
Names that contain "map_plit" contain stage lighting information.

And so on....

i'll see if i can outline some of the structures in the format you specified. Assuming you don't beat me to it, heh.

There are a number of root node names that are not currently known, and thus structure areas from multiple files need to be compared in order to obtain the correct size and other values contained within them.

This is where the relocation table becomes useful in its role for validation of file offsets.

Without the aid of the relocation table, there would be no way to tell if a value in the data section is actually an offset or just a count, or some other random set of values. The way the relocation table helps is that if there is an offset to a location specified in the relocation table, the you automatically know it is a file offset value, without question.

As an example, let say you see a value a some arbitrary location, 0x3BC, for instance. And lets say the value at that location is 0x4C.

Without further information, you have no idea what this value represents. It could be a count, it could be a size, it could be flags, etc.

But if in the relocation table you find an offset values of 0x39C (remember offsets are relative to the data section, so adding 0x20 would make this offset become 0x3BC), you now know that the value at that location is definitely and offset value. With that knowledge you can be confident in the fact that if you were to add 0x20 to 0x4C to become 0x6C, that there would be another structure or set of values at that location.

That is the purpose of the relocation table...to know every location that is a valid file offset in the data section.

As another example, say you have a structure that you already know contains a offset to something, but the value you currently see happens to be 0x00000000. how do you know if this is actually a valid offset (that would become 0x20) or if it is simply an invalid or NULL offset, and thus is supposed to be 0? Without the relocation table, you would not be able to tell the difference. Checking the relocation table if there is an offset to a location with a value, even if it is 0 or something would never think was an offset, you can be certain that is in fact a file offset value and should be modified accordingly (adding 0x20).


The proper use of the relocation table is to go through every value in it, and for ever value at the offset it specifies, add 0x20 to it.

Code:
int NUM = 0; // our counter value
while NUM is less relocation count
	goto relocation offset [NUM] + 0x20
	add 0x20 to value at this location
	increment NUM
After this is done, even zero (0) value offsets, would become +0x20. As such when you have a known structure, you can be sure that the offset value is valid.

So proper parsing of every dat/usd file comes down to figuring out all the various structures used within them.

Hopefully that helped.
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
I'm gonna have to read into that alot more...

must I say that you are way too descriptive when it comes to data :p
I use math instead of words when I describe something... :/
(not trying to be offencive if it sounds like it to you)

but yea...
umm...

nice find on the reference node btw :o

I still don't understand how to get to the offset I need :p

maybe I'm just missing something...

I'm thinking the root nodes just point to specific blocks of data and change the way you read the data after passing one...
^that prbly made no sence :p

what I mean iswhen you're reading the data from the offset and the offset hits a root node,
the way you read the offsets from the relocation table then changes...

am I right on that??
 

revel8n

Smash Cadet
Joined
Dec 1, 2009
Messages
51
Unless you want to validate a file offset, or when you get to a point where you understand how to add, move, or otherwise modify data file size, or structure locations...
You can ignore the relocation table.
The purpose of the relocation table is for locating file offsets in the data section.
You cannot tell any structure information from the relocation table alone.

So, to take an example from your favorite file, PlPkNr, heh:

Here is the file header:
Code:
00 02 11 D2 00 01 FF 80 00 00 04 7A 00 00 00 02
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Segmented out:
Code:
00 02 11 D2 - 0x00 - file size
00 01 FF 80 - 0x04 - data section sixe
00 00 04 7A - 0x08 - relocation entry count
00 00 00 02 - 0x0C - root node count
00 00 00 00 - 0x10 - reference node count
00 00 00 00 - 0x14 - ??
00 00 00 00 - 0x18 - ??
00 00 00 00 - 0x1C - ??
So a file of size 0x211D2, which can be compared against the reported size for validation, or to tell if there is more data that follows this file data (As in Pl**Aj.dat files)...

The data section is 0x1FF80 bytes in length.

There are 0x47A (1146) relocation table entries, making the relocation table 0x11E8 bytes in length.

There are 2 root node, taking up 16 bytes.

There are 0 reference nodes.

The header is 0x20 bytes in length so the data section starts at 0x20.

The data section is 0x1FF80 bytes in length so the relocation table starts at offset 0x1FFA0 = data start (0x20) + data size (0x1FF80).

The rrelocation table is 0x11E8 bytes in length so the root nodest start at 0x1FFA0 + 0x11E8 = 0x21188.

The root nodes take up 0x10 bytes so the reference nodes would have started at 0x21198.

The reference nodes take up 0 bytes of space, so instead the string table starts at 0x21198.

So no all starting offsets and initial section sizes are known.

For the moment the relocation table can otherwise be ignored, so starting with the root node data:
Code:
00 01 2B 88 00 00 00 00 00 01 7F E8 00 00 00 19
Given we know the root/reference node structures are 8-bytes each, containing 2 4-byte offsets, and the file header states there are 2 of them, we have:
Code:
root node [0]
00 01 2B 88 - data offset
00 00 00 00 - string table offset

root node [1]
00 01 7F E8 - data offset
00 00 00 19 - string table offset
Looking at this, there is still no direct indicator as to what each rot node represents.

We know the string table starts at 0x21198, so let see what there:
Code:
50 6C 79 50 69 6B 61 63 68 75 35 4B 5F 53 68 61
72 65 5F 6A 6F 69 6E 74 00 50 6C 79 50 69 6B 61
63 68 75 35 4B 5F 53 68 61 72 65 5F 6D 61 74 61
6E 69 6D 5F 6A 6F 69 6E 74 00
Which in ASCII translates to:

Code:
PlyPikachu5K_Share_joint\0PlyPikachu5K_Share_matanim_joint\0
Looking at the two string table offsets of 0x0000000 and 0x00000019 and using them as offsets from teh beginning of the string table you can see that:
root node 0 is named "PlyPikachu5K_Share_joint"
root node 1 is named "PlyPikachu5K_Share_matanim_joint"

Studying the root node names, most of them are the same, and given that we know the file deals with Pikachu, the beginning can be ostly ignore as simply telling us the name of the character.

The real parts of interest is at the end of these strings, name "_joint" and "_matanim_joint".

The "_joint" alone specifies that the root node data offset deals with a joint hierarchy and its related information. The second has "_matanim" and for now is assumed to deal with material animations (unconfirmed), and will be ignored for now.

Going back to root node 0, its data offset is: 0x12B88, which is relative to the data section so adding 0x20 would be 0x12BA8.

So at offset 0x12BA8 should be located a "_joint" data structure.

We know this structure as:
Code:
struct JOBJ_DATA
{
	// 0x00
	uint32 unknown0x00       <format = hex>;
	uint32 flags             <format = hex>;
	uint32 childOffset       <format = hex>; // child jobj structure
	uint32 nextOffset        <format = hex>; // next jobj structure
	// 0x10
	uint32 dobjOffset        <format = hex>; // dobj structure - object information?
	float3 rotation;                         // rotation
	float3 scale;                            // scale
	float3 translation;                      // translation
	uint32 transformOffset   <format = hex>; // inverse transform
	uint32 unknown0x3C;
	// 0x40
};
Which translates into:
Code:
0x00 (4): ??
0x04 (4): joint flags
0x08 (4): offset to child joint in hierarchy (+ 0x20)
0x0C (4): offset to sibling joint in hierarchy (+ 0x20)
0x10 (4): offset to data object structure (+0x20)
0x14(12): 3 float values representing joint rotation
0x20(12): 3 float values representing joint scale
0x2C(12): 3 float values representing joint translation
0x38 (4): offset to inverse joint transform (+ 0x20)
0x3C (4): ??
So when you find a root node whose name contains "_joint" without "_matanim" or "_shapeanim", you will find a structure matching the above layout when you follow the data offset to its location.

The joints are a hierarchy of information so following the offsets contained within it lead to structures of the same type as well as others that will allow you to gain access to the geometry and material data.

The joint object structure above, and its translation values, are the main structures being modified in some of the hacks posted by Milun (thus the C0/40 confusion) that move various elements around.
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
ah...

you were explaining it to me in a completely different POV than what I needed...
which is why I misunderstood everything you were telling me about finding the data...

I get the POV you're explaining to me now...

but that doesn't really tell me how to get what data I need...

yes I see that it offsets to joint (bone) data, but yea...
does it continue reading on from there??

I can't think up how to build the correct equation for the dat file

but I used to put the relocation table ahead of the root nodes...

so what,
you read the root nodes first and then get the offsets to the relocation table after that??

you're not really telling me how I can merge the relocation offsets,
(which takes you to needed data from those offsets),
with the root nodes which explain what type of data you're looking at,
and what type of method used to read the data...

^that much I have known since GF explained the relocated data offsets which located somewhere else in the data block...

the only thing I don't know how to find is how the cpu can tell what data it's looking at...

I can't really give a solid equation until I know how to build one...

now, I've gotten as far as to seperate the relocation offsets and root nodes with their names...
^using a 'FOR' function for the offsets and root nodes...

what I need to do next is somehow tell the template what data it's reading so it know what method to use...

now I know the relocation table alone won't do that...
^I've known that since emutalk...

the root nodes work similar to the relocation table,
but offset to a particular spot in the data block

the relocation offsets may have something to play when going through the data offsets before deciding a method to use...
(only reading the data offset which offsets to either before or after a root node location)
^I've know that since GF explained that as well...
although I had a similar idea to that from before,
when I thought the relocation offsets were the data offsets...

anyways...
so I'm thinking,
if the offset is before this root node, use this conversion method...
if the offset is after this root node, then use this conversion method...

that's where I have currently been standing since I posted my first post here...

and now that I've finally figured out a way to explain myself :glare:
hopefully I can get further than this...

would you like to see my template in it's current state??
 

revel8n

Smash Cadet
Joined
Dec 1, 2009
Messages
51
For now, ignore the relocation table in terms finding data. You cannot tell what the data is from the relocation table.

You can ONLY tell that something is an offset from the relocation table.

What i am trying to show through the root node usage is that:
For Pl an Ty files for example, when you see a "_joint" root node, you know for a fact that the structure the data offset leads to follows the JOBJ structure format.

From that structure you have more offsets to more JOBJ structures (The values at 0x08 and 0x0C - Add 0x20 to those and you will find JOBJ data at those offsets), as well as an offset to a DOBJ structure (the value at 0x10 in the JOBJ structure - Add 0x20 and follow that offset and it will lead to a DOBJ structure).

The DOBJ structure contains offsets to more DOBJ structures as well as offsets to both material (MOBJ) and geometry structures (GOBJ).

The MOBJ structures have offsets to material color and texture (TOBJ) structures.

The TOBJ structure contains offsets to both IMAGE and PALETTE headers.

The GOBJ structure has offsets to mesh and vertex attribute structures.

The structures form a hierarchy of data, flags, and offsets to other data structures.

Once you find a root node that that has a structure you know for sure it starts with you simply follow the hierarchy of structure, their data, and offsets.
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
hmm...
I see...
so the "tree" starts at the joints, and works all the way down to the textures...

heh...
you know...

I feel like I'm being taken by the arm from my high place of authority,
and being led to some place that will better benefit me... XDD

I hope this works man :)
and I hope to have a working :/

I'm not worried at all cause I have the feeling it will :)

EDIT:
hey revel...
when it comes time tomorrow,
I'd like to start from scratch to see if I can't get a working template...

all I need is your guidence to any Q's I may have...
(which will prbly be alot)

we could better take care of this on my chat :)
^this can be found here: http://tcll5850.webfreehosting.net
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
alright! :D
now I'm finally getting somewhere >:)



^that look about right to you Revel??

note:
the OBJ structure definition is what deciphers what data it's reading based on the node string...
it then chooses the corresponding *OBJ structure definition when reading the data...

EDIT:
btw, I think the '_share' stands for library data, but I'm not for certain :/
 
Last edited:

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
KK...
so now there's just a few things left to note...

how do you tell when the bone tree data ends and the DOBJ data starts??
____what does the 'D' stand for?? :p *Derp-ing*

how do you tell the end of the data tree branches??


on an off-note:
what's the point of the relocation table??

I mean... I know it's for editing the data and such, but how??
^what "role" does it play??

EDIT:
O.o
wait...
would the offsets from the relocation offset data offset to the stop points of the data tree branches??
^just a wild guess there :/

EDIT2:
wanna meet me on my chat now :p
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
O.O

IDK if Revel's found this yes...
but I've noticed that the MOBJ data in Revel's POV of the file,
offsets to a certain part in 1 of the 5 objects in the first section of the relocation table...

I think there is a stop section located somewhere when you hit the 5th offset in the first section that bumps it to the next section which is read a different way...
but I'll have to keep looking to answer that ;)

I'm going along with what Revel told me and with my knowledge I've found out about the relocation table...

I'm getting somewhere...
but just logging where I am atm :)

I'll prbly be much further when someone replies here :p

and btw,
has the shader data been figured out yet??
and there should also be pallets (maybe)...
I'll answer that one myself :|

EDIT:
pallets are true :p
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
so, hey...

once I've got the tree, how do I use it??

the relocation table keeps bugging me,
and I'm just curious to see how it fits in with the structures :/

since I know the table deals with seperate indexes,
and the first offsets in the table follow to *trying to rmbr* I think object data blocks...

IDK, I'll prbly implement both my offset method,
and yur tree method into my tamplate...

I think I actually may need the table in order to build the full tree...
IDK how to add seperate bones from the parent offsets to work in the template :/
it only works for the childs right now...

maybe if I use an 'IF' statment *thinking*

I'm at skool right now...
I'm just trying to research this as I'm working on my other projects...
(mainly my viewer and converter)

I sooo hate fame :glare:

I've got alot to talk about since you intro'd this "new" way of finding the data...
but IDK where to start :p

once I get home,
will you meet me on my chat so we can discuss this :/
I'm just so confused be switching methods... :urg:

EDIT:
actually, not tonight...
I just rmbr'd I had a friend brawl tonight :/
...tomorrow then :)
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
hey Revel...

can I get your current src for your viewer??

I'm getting alot better at reading C++
and sending me a well commented src would help to explain things alot :)

what I mean by 'well commented' is every line has a comment telling what it does

would you mind doing that
(doesn't have to be that well commented (I can do some figuring))
^it'd just be convenient is all :)

EDIT:
I have the src...
but for the program, I'm getting the 'SideBySide' error...

I noticed the reference to the assembly 'Microsoft.VC90.DebugCRT'
I'm looking into this... but it's causing the config error

and I also found that MSVCR90D.dll is required after using Resource Hacker to comment out the assembly :p

EDIT2:
Visual Studio 2008 is required

EDIT3:
just reported the MPXviewer fix :)
it's simple, but unconveinient...
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
ignoring all of my previous statements for which I have alot assumed and answered now...

I believe the relocation table may be for indexing strings (names) to certain offsets...
^(from a local file containing the strings)
the strings would have to be in the same order as the offset list...
^(I'll do a little more research on this)

the heirarchy of data that revel explains from the root nodes is how you'd edit the data...
(you'd have tor re-write the entire heirarchy)

but now 1 small Q I have is:
what's the significance of the '_joint' after the '_matanim' in the 2nd root node of Pikachu's data??
could it possibly be a method of parenting the "animation" to the object??

EDIT:
I'm renaming the relocation table...
it should be called 'Offset Table'

the offset table (from what I'm finding) seems to have nothing to do with editing...
it seems to either group/pair data, or perhapse index strings/names to certain locations

I'm adding an option to compair the current position in the heirarchy,
with the offsets in the table...

I should get something out of that :/
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
btw,
a little something on 3D terms:

I noticed Nintendo messed up the 3D terms, to make some of their own

Vertex = Facepoint
Polygon = Face

also, IDK if these exist here,
but the wii uses TEV's which are basically an extension of the materials

the TEV's define the shaders used by the materials
(basically the TEV's are the shaders, for the shader settings of the materials)
it's confusing... I know

another thing...
do facepoints have the same structure used by the CP and XF registers as they do with the wii formats??
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
got into looking at melee's dat files a little more

I coulda been told this honestly... -_-
but it looks like the offset table goes in comparison with the root node offset's offsets

all of the offsets you encounter from the root nodes,
can also be encountered from the offset table

so the offset table can either be a way to orginize the data,
a way to send strings to specific offsets,
or a way to output data...

I think I'm gonna go with the strings for my guess,
but my Q is:
where would I find the string lists for the offset table??
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
I've looked into the crappy YAGCD again
(knowing alot more about it's docs)

CP and XF registers are the same,
meaning facepoints can have a max of 20 indexes

if you need more info, you can refer to the MDL0 resource thread on KCMM:
http://forums.kc-mm.com/index.php?topic=19452.0

I've logged alot of info on the facepoint indexes,
and the commands should be the exact same...
(YAGCD actually lists more than what I've put up)

I'm gonna have to make a plugin for a few plugins of mine
this will shorten the data usage of my MDL0 plugin,
as well as allow acces from the dat plugin...
(not sure how to code the functions for this yet)

Revel8n
where are you :(
I miss progging with you :c
 

revel8n

Smash Cadet
Joined
Dec 1, 2009
Messages
51
i am still around, been rather busy.

As far as the relocation/offset table is concerned:
- It is nothing more than just that, a table listing the offsets to every offset value in the file.
- It is a programming aid that allows the file to be used "as is" within the game.
- The table is used to update all the offsets within the file with the actual addresses in memory.
- If there is a location referenced in the reloc/offset table, then you know for sure that the value is actually a valid offset, even if it is 0 or some other weird value.
- There is nothing else you can tell from the reloc/offset table than the facts above.
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
hmm...

wat U been doin *curious* :p
I saw a few of your posts about the dat format on a few other forums...
notice you've been working with RTB who has left me on the mdl0 format for some reason >_>

eh...
was trying to find any new info that would help me put together a model...

so with the exception of melee,
I guess I'd have to customly name each bone,
since bones are referenced by a number, and apparently there is no name strings :/

I'll have to make a file for that,
but first I'll work on naming bones by their ID's

and my format requires string names:
< bone name position(X,Y_grid_coords) is_connected(bool) locX,Y,Z rotX,Y,Z scaX,Y,Z bind inv_bind >
< bone Top 0,0 0 0,0,0 0,0,0 1,1,1 1,0,0,0|0,1,0,0|0,0,1,0 1,0,0,0|0,1,0,0|0,0,1,0 >

btw,
think you could give me a hand with my format,
and maybe even explain some 3D info to me that I'm missing??

also, I'm having a problem with MDL0 vert indexing on facepoints...
think you might be able to help me out there??
I think it may have something to do with the XF register
but IDK...
maybe I'm just forgetting to do something in the CP register
or maybe I'm handelling the code wrong...
uugh... so many possibilities... so little time...
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
also, how to go about handelling the offset table when comparing offsets,

would I simply put the offsets in a list??
I'm thinking to create list indexes to put together the seperate parts
(or rather break them apart)

note: not every offset in the table links to an offset,
some have linked to data, such as texture w/h

so you could basically create offset groups w/in the table,
and change the group-type based on offset position
but I have no idea how to even do that XD

how did you get the models to show...
I've tried MPX_viewer, and that doesn't work alone...
meaning you have a src somewhere

may I have that src :/
my C skills have improved, but I'm still learning

I should prbly get caught up to what 'm talking about XD
I'm way too far ahead of myself... heh
 

Tcll

Smash Lord
Joined
Jul 10, 2010
Messages
1,780
Location
The Gates of Darkness
NNID
Tcll5850
hmm...
hey revel...
you never mentioned how you figured out the reference nodes, or how to use them :/

I'm trying to rename everything to sound a bit simpler

root-nodes -> Base Offsets
ref-nodes -> ?? (IDK what these do)
 

tera twin

Smash Journeyman
Joined
Nov 26, 2010
Messages
280
Required self bumping because nobody cares about this hidden gem of a thread.
 
Top Bottom