-THIS IS INCOMPLETE! I'M ONLY COMMITING IT IN ORDER TO SOLICIT COMMENTS
-FROM A FEW PEOPLE. DON'T TAKE THIS AS THE FINAL VERSION YET.
+THIS IS INCOMPLETE! I'M ONLY COMMITING IT IN ORDER TO SOLICIT COMMENTS
+FROM A FEW PEOPLE. DON'T TAKE THIS AS THE FINAL VERSION YET.
Samba4 Programming Guide
Finally, I think that for code that is parsing or formatting protocol
packets the code layout should strongly reflect the packet
format. That means ordring the code so that it parses in the same
-order as the packet is stored on the while (where possible) and using
+order as the packet is stored on the wire (where possible) and using
white space to align packet offsets so that a reader can immediately
map any line of the code to the corresponding place in the packet.
"text"). So you can have constant tables of protocol data.
-Memory Contexts
----------------
+How to use talloc
+-----------------
-We introduced the talloc() system for memory contexts during the 2.2
-development cycle and it has been a great success. It has greatly
-simplified a lot of our code, particularly with regard to error
-handling.
-
-In Samba4 we use talloc even more extensively to give us much finer
-grained memory management. The really important thing to remember
-about talloc in Samba4 is:
-
- "don't just use the first talloc context that comes to hand - use
- the RIGHT talloc context"
-
-Just using the first talloc context that comes to hand is probably the
-most common systematic bug I have seen so far from programmers that
-have worked on the Samba4 code base. The reason this is vital is that
-different talloc contexts have vastly different lifetimes, so if you
-use a talloc context that has a long lifetime (such as one associated
-with a tree connection) for data that is very short lived (such as
-parsing an individual packet) then you have just introduced a huge
-memory leak.
-
-In fact, it is quite common that the correct thing to do is to create
-a new talloc context for some little function and then destroy it when
-you are done. That will give you a memory context that has exactly the
-right lifetime.
-
-You should also go and look at a new talloc function in Samba4 called
-talloc_steal(). By using talloc_steal() you can move a lump of memory
-from one memory context to another without copying the data. This
-should be used when a backend function (such as a packet parser)
-produces a result as a lump of talloc memory and you need to keep it
-around for a longer lifetime than the talloc context it is in. You
-just "steal" the memory from the short-lived context, putting it into
-your long lived context.
+Please see the separate document, source/lib/talloc/talloc_guide.txt
+You _must_ read this if you want to program in Samba4.
Interface Structures
In Samba3 many of the core wire structures in the SMB protocol were
never explicitly defined in Samba. Instead, our parse and generation
functions just worked directly with wire buffers. The biggest problem
-with this is that is tied our parse code with out "business logic"
+with this is that is tied our parse code with our "business logic"
much too closely, which meant the code got extremely confusing to
read.
are extremely important differences in the approach that are worth
pointing out.
-In the Samba3 msrpc code we used explicit parse strucrures for all
+In the Samba3 msrpc code we used explicit parse structures for all
msrpc functions. The problem is that we didn't just put all of the
real variables in these structures, we also put in all the artifacts
as well. A good example is the security descriptor strucrure that
*NOT* be in the interface structure.
In Samba3 there were unwritten rules about which variables in a
-strucrure a high level caller has to fill in and which ones are filled
+structure a high level caller has to fill in and which ones are filled
in by the marshalling code. In Samba4 those rules are gone, because
the redundent artifact variables are gone. The high level caller just
sets up the real variables and the marshalling code worries about
then in your callback function you can call the smb_raw_XXXX_recv()
function to receive the reply. Your callback will receive the "req"
-pointer, which you can use to retrieve your private data.
+pointer, which you can use to retrieve your private data from
+req->async.private.
Then all you need to do is ensure that the main loop in the client
library gets called. You can either do that by polling the connection
notify) often don't return for hours, while hundreds of other
functions pass through the socket. Take a look at the RAW-MUX test in
the Samba4 smbtorture to see some really extreme examples of the sort
-of async operations that Windows supports.
+of async operations that Windows supports. I particularly like the
+open/open/close sequence where the 2nd open (which conflicts with the
+first) succeeds because the subsequent close is answered out of order.
In Samba3 we handled this stuff very badly. We had awful "pending
request" queues that allocated full 128k packet buffers, and even with
all that crap we got the semantics wrong. In Samba4 I intend to make
sure we get this stuff right.
-So, how do we do this? We now an async interface between smbd and the
-NTVFS backends. Whenever smbd calls into a backend the backend has an
-option of answer the request in a synchronous fashion if it wants to
-just like in Samba3, but it also has the option of answering the
+So, how do we do this? We now have an async interface between smbd and
+the NTVFS backends. Whenever smbd calls into a backend the backend has
+an option of answer the request in a synchronous fashion if it wants
+to just like in Samba3, but it also has the option of answering the
request asynchronously. The only backend that currently does this is
the CIFS backend, but I hope the other backends will soon do this to.
that tells smbd that the backend has elected to reply later rather
than replying immediately. The backend must *only* do this if
req->async.send_fn is not NULL. If send_fn is NULL then it means that
-smbd cannot handle this function being replied to in an async fashion.
+the smbd front end cannot handle this function being replied to in an
+async fashion.
If the backend does this then it is up to the backend to call
req->async.send_fn() when it is ready to reply. It the meantime smbd
function, so smbd has a _send() function and the parse function for
each SMB.
-Go and have a look at reply_getatr_send() and reply_getatr() in
-smbd/reply.c. Read them? Good.
+As an example go and have a look at reply_getatr_send() and
+reply_getatr() in smb_server/reply.c. Read them? Good.
Notice that reply_getatr() sets up the req->async structure to contain
-the send function. Thats how the backend gets to do an async
-reply. Also notice that reply_getatr() only does the parsing of the
-request, and does not to the reply generation. That is done by the
-_send() function. Nice and simple really.
+the send function. Thats how the backend gets to do an async reply, it
+calls this function when it is ready. Also notice that reply_getatr()
+only does the parsing of the request, and does not do the reply
+generation. That is done by the _send() function.
+
+The only missing piece in the Samba4 right now that prevents it being
+fully async is that it currently does the low level socket calls (read
+and write on sockets) in a blocking fashion. It does use select() to
+make it somewhat async, but if a client were to send a partial packet
+then delay before sending the rest then smbd would be stuck waiting
+for the second half of the packet.
+
+To fix this I plan on making the socket calls async as well, which
+luckily will not involve any API changes in the core of smbd or the
+library. It just involves a little bit of extra code in clitransport.c
+and smbd/request.c. As a side effect I hope that this will also reduce
+the average number of system calls required to answer a request, so we
+may see a performance improvement.
+
+
+NTVFS
+-----
+
+One of the most noticeable changes in Samba4 is the introduction of
+the NTVFS layer. This provided the initial motivation for the design
+of Samba4 and in many ways lies at the heart of the design.
+
+In Samba3 the main file serving process (smbd) combined the handling
+of the SMB protocol with the mapping to POSIX semantics in the same
+code. If you look in smbd/reply.c in Samba3 you see numerous places
+where POSIX assumptions are mixed tightly with SMB parsing code. We
+did have a VFS layer in Samba3, but it was a POSIX-like VFS layer, so
+no matter how you wrote a plugin you could not bypass the POSIX
+mapping decisions that had already been made before the VFS layer was
+called.
+
+In Samba4 things are quite different. All SMB parsing is performed in
+the smbd front end, then fully parsed requests are passed to the NTVFS
+backend. That backend makes any semantic mapping decisions and fills
+in the 'out' portion of the request. The front end is then responsible
+for putting those results into wire format and sending them to the
+client.
+
+Lets have a look at one of those request structures. Go and read the
+definition of "union smb_write" and "enum write_level" in
+include/smb_interfaces.h. (no, don't just skip reading it, really go
+and read it. Yes, that means you!).
+
+Notice the union? That's how Samba4 allows a single NTVFS backend
+interface to handle the several different ways of doing a write
+operation in the SMB protocol. Now lets look at one section of that
+union:
+
+ /* SMBwriteX interface */
+ struct {
+ enum write_level level;
+
+ struct {
+ uint16 fnum;
+ SMB_BIG_UINT offset;
+ uint16 wmode;
+ uint16 remaining;
+ uint32 count;
+ const char *data;
+ } in;
+ struct {
+ uint32 nwritten;
+ uint16 remaining;
+ } out;
+ } writex;
+
+see the "in" and "out" sections? The "in" section is for parameters
+that the SMB client sends on the wire as part of the request. The smbd
+front end parse code parses the wire request and fills in all those
+parameters. It then calls the NTVFS interface which looks like this:
+
+ NTSTATUS (*write)(struct request_context *req, union smb_write *io);
+
+and the NTVFS backend does the write request. The backend then fills
+in the "out" section of the writex structure and gives the union back
+to the front end (either by returning, or if done in an async fashion
+then by calling the async send function. See the async discussion
+elsewhere in this document).
+
+The NTVFS backend knows which particular function is being requested
+by looking at io->generic.level. Notice that this enum is also
+repeated inside each of the sub-structures in the union, so the
+backend could just as easily look at io->writex.level and would get
+the same variable.
+
+Notice also that some levels (such as splwrite) don't have an "out"
+section. This happens because there is no return value apart from a
+status code from those SMB calls.
+
+So what about status codes? The status code is returned directly by
+the backend NTVFS interface when the call is performed
+synchronously. When performed asynchronously then the status code is
+put into req->async.status before the req->async.send_fn() callback is
+called.
+
+Currently the most complete NTVFS backend is the CIFS backend. I don't
+expect this backend will be used much in production, but it does
+provide the ideal test case for our NTVFS design. As it offers the
+full capabilities that are possible with a CIFS server we can be sure
+that we don't have any gaping holes in our APIs, and that the front
+end code is flexible enough to handle any advances in the NT style
+feature sets of Unix filesystems that make come along.
+
+
+Process Models
+--------------
+
+In Samba3 we supported just one process model. It just so happens that
+the process model that Samba3 supported is the "right" one for most
+users, but there are situations where this model wasn't ideal.
+
+In Samba4 you can choose the smbd process model on the smbd command
+line.
+
+
+DCERPC binding strings
+----------------------
+
+When connecting to a dcerpc service you need to specify a binding
+string.
+
+The format is:
+
+ TRANSPORT:host[flags]
+
+where TRANSPORT is either ncacn_np for SMB or ncacn_ip_tcp for RPC/TCP
+
+"host" is an IP or hostname or netbios name. If the binding string
+identifies the server side of an endpoint, "host" may be an empty
+string.
+
+"flags" can include a SMB pipe name if using the ncacn_np transport or
+a TCP port number if using the ncacn_ip_tcp transport, otherwise they
+will be auto-determined.
+
+other recognised flags are:
+
+ sign : enable ntlmssp signing
+ seal : enable ntlmssp sealing
+ spnego : use SPNEGO instead of NTLMSSP authentication
+ krb5 : use KRB5 instead of NTLMSSP authentication
+ connect : enable rpc connect level auth (auth, but no sign or seal)
+ validate : enable the NDR validator
+ print : enable debugging of the packets
+ bigendian : use bigendian RPC
+ padcheck : check reply data for non-zero pad bytes
+
+
+Here are some examples:
+
+ ncacn_np:myserver
+ ncacn_np:myserver[samr]
+ ncacn_np:myserver[\pipe\samr]
+ ncacn_np:myserver[/pipe/samr]
+ ncacn_np:myserver[samr,sign,print]
+ ncacn_np:myserver[sign,spnego]
+ ncacn_np:myserver[\pipe\samr,sign,seal,bigendian]
+ ncacn_np:myserver[/pipe/samr,seal,validate]
+ ncacn_np:
+ ncacn_np:[/pipe/samr]
+ ncacn_ip_tcp:myserver
+ ncacn_ip_tcp:myserver[1024]
+ ncacn_ip_tcp:myserver[sign,seal]
+ ncacn_ip_tcp:myserver[spnego,seal]
+
+
+IDEA: Maybe extend UNC names like this?
+
+ smbclient //server/share
+ smbclient //server/share[sign,seal,spnego]
+
+DCERPC Handles
+--------------
+The various handles that are used in the RPC servers should be created and
+fetch using the dcesrv_handle_* functions.
+
+Use dcesrv_handle_new(struct dcesrv_connection *, uint8 handle_type) to obtain
+a new handle of the specified type. Handle types are unique within each
+pipe.
+The handle can later be fetched again using
+struct dcesrv_handle *dcesrv_handle_fetch(struct dcesrv_connection *dce_conn, struct policy_handle *p, uint8 handle_type)
+and destroyed by dcesrv_handle_destroy(struct dcesrv_handle *).
+
+User data should be stored in the 'data' member of the dcesrv_handle struct.
MSRPC
- msrpc
+don't zero structures! avoid ZERO_STRUCT() and talloc_zero()
GMT vs TZ in printout of QFILEINFO timezones
test timezone handling by using a server in different zone from client
-don't just use any old TALLOC_CTX, use the right one!
-
do {} while (0) system
NT_STATUS_IS_OK() is NOT the opposite of NT_STATUS_IS_ERR()
need to implement secondary parts of trans2 and nttrans in server and
client
-add talloc_steal() to move a talloc ptr from one pool to another
-
document access_mask in openx reply
check all capabilities and flag1, flag2 fields (eg. EAs)
conference paper
developer docs
+
+svn instructions
+
+Ideas
+-----
+
+ - store all config in config.ldb
+
+ - load from smb.conf if modtime changes
+
+ - dump full system config with ldbsearch
+
+ - will need the ability to form a ldif difference file
+
+ - advanced web admin via a web ldb editor
+
+ - normal web admin via web forms -> ldif
+
+ - config.ldb will replace smb.conf, secrets.tdb, shares.tdb etc
+
+ - subsystems in smbd will load config parameters for a share
+ using ldbsearch at tconx time
+
+ - need a loadparm equivalent module that provides parameter defaults
+
+ - start smbd like this: "smbd -C tdb://etc/samba/config.ldb" or
+ "smbd -C ldapi://var/run/ldapi"
+
+ - write a tool that generates a template ldap schema from an existing
+ ldb+tdb file
+
+ - no need to HUP smbd to reload config
+
+ - how to handle configuration comments? same problem as SWAT
+
+
+BUGS:
+ add a test case for last_entry_offset in trans2 find interfaces
+ conn refused
+ connect -> errno
+ no 137 resolution not possible
+ should not fallback to anon when pass supplied
+ should check pass-thu cap bit, and skip lots of tests
+ possibly allow the test suite to say "allow oversized replies" for
+ trans2 and other calls
+ handle servers that don't have the setattre call in torture
+ add max file coponent length test and max path len test
+ check for alloc failure in all core reply.c and trans2.c code where
+ allocation size depends on client parameter
+
+case-insenstive idea:
+ all filenames on disk lowercase
+ real case in extended attribute
+ keep cache of what dirs are all lowercase
+ when searching for name, don't search if dir is definately all lowercase
+ when creating file, use dnotify to tell if someone else creates at
+ same time
+
+solve del *.* idea:
+ make mangle cache dynamic size
+ fill during a dir scan
+ setup a timer
+ destroy cache after 30 sec
+ destroy if a 2nd dir scan happens on same dir
+
git.samba.org / samba.git / blobdiff