<[sambabg] ==== ====[plain] <[center] \Large '''Multi-Channel''' [center]> [frame]> [sambabg]> ==== Multi-Channel - General ==== <[block]{multiple transport connections in one SMB(3) session} * '''channel''': transport connection bound to a session * client decides which connections to bind and to use * session is valid as long as at least one channel is intact [block]> <[block]{two purposes} # increase throughput: #* use multiple connections of same type # improve fault tolerance: #* channel failure: replay/retry detection %%#* session is valid as long as one channel is still intact [block]> ==== Multi-Channel - General ==== <[block]{use case: channels of different type/quality} * use only the channels of best quality * fall back to inferior channels if superior ones fail * e.g.: laptop switching between WiFi and LAN (?) [block]> ==== Multi-Channel - Windows/Protocol ==== # establish initial session on TCP connection # find interfaces with interface discovery: \\ % @FSCTL\_QUERY\_NETWORK\_INTERFACE\_INFO@ # bind additional TCP (or later RDMA) connection (channel) to established SMB3 session (''session bind'') # Windows: uses connections of same (and best) quality # Windows: binds only to a single node # replay / retry mechanisms, epoch numbers ==== Multi-Channel $\in$ Samba ==== <[block]{samba/smbd: multi-process} * '''Originally:''' process $\Leftrightarrow$ TCP connection *<0> '''Idea:''' transfer new TCP connection to existing smbd *<0> '''How?''' ==> use fd-passing (sendmsg/recvmsg) *<0> '''When?''' ** ''Natural choice'': at SessionSetup (Bind) ** !Idea!: as early as possible, based on ClientGUID \\ % ==> per ClientGUID single process model %%*<0> !But!: This may not work! \frownie [block]> ==== Multi-Channel $\in$ Samba ==== <[center] <<>> [center]> ==== Multi-Channel $\in$ Samba ==== <[block]{samba/smbd: multi-process} * '''Originally:''' process $\Leftrightarrow$ TCP connection * '''Idea:''' transfer new TCP connection to existing smbd *<0> '''How?''' ==> use fd-passing (sendmsg/recvmsg) *<0> '''When?''' ** ''Natural choice'': at SessionSetup (Bind) ** !Idea!: as early as possible, based on ClientGUID \\ % ==> per ClientGUID single process model %%*<0> !But!: There may be problems! ... \frownie [block]> ==== Multi-Channel $\in$ Samba ==== -<1>{ <[center] <<>> [center]> } -<2>{ <[center] <<>> [center]> } -<3>{ <[center] <<>> [center]> } ==== Multi-Channel $\in$ Samba ==== <[block]{samba/smbd: multi-process} * '''Originally:''' process $\Leftrightarrow$ TCP connection * '''Idea:''' transfer new TCP connection to existing smbd * '''How?''' ==> use fd-passing (sendmsg/recvmsg) *<2-> '''When?''' ** ''Natural choice'': at SessionSetup (Bind) ** !Idea!: as early as possible, based on ClientGUID \\ % ==> per ClientGUID single process model %%*<0> !But!: There may be problems! ... \frownie [block]> ==== Multi-Channel $\in$ Samba : pass by ClientGUID ==== <[center] %%<<>> <<>> [center]> ==== Multi-Channel $\in$ Samba : pass by ClientGUID ==== <[block]{Wait a minute - what about performance?} * Single process... * But we use short-lived worker-pthreads for I/O ops! * Benchmarks and tunings still to be done. [block]> %%%% ==== Multi-Channel $\in$ Samba ==== %%%% %%%% <[block]{samba/smbd: multi-process} %%%% * '''Orignally:''' process $\Leftrightarrow$ TCP connection %%%% * '''Idea:''' transfer new TCP connection to existing smbd %%%% * '''How?''' ==> use fd-passing (sendmsg/recvmsg) %%%% * '''When?''' %%%% ** ''Natural choice'': at SessionSetup (Bind) %%%% ** ''Idea'': as early as possible, based on ClientGUID \\ % %%%% ==> per ClientGUID single process model %%%% * !But!: There may be problems! ... \frownie %%%% [block]> %%%% %%%% %%%% ==== The Relevance of the ClientGUID ==== %%%% %%%% +<1->{ %%%% <[block]{Assumption was:} %%%% * All channels in a session have the same ClientGUID %%%% * The server enforces this %%%% [block]> %%%% } %%%% %%%% +<2->{ %%%% <[block]{Evidence from [MS-SMB2]:} %%%% * 3.3.5.9 Receiving an SMB2 CREATE Request: %%%% ** sets Open.ClientGuid to Connection.ClientGuid %%%% ** replay detection checks \\ % %%%% Open.ClientGuid == Connection.ClientGuid %%%% * 3.3.5.9.7/12 Durable (v2) Reconnect Create Context: %%%% ** check Open.ClientGuid == Connection.ClientGuid %%%% [block]> %%%% } %%%% %%%% +<3->{ %%%% <[block]{The truth is...} %%%% The Windows server does not enforce it! %%%% [block]> %%%% } %%%% %%%% %%%% ==== The Relevance of the ClientGUID ==== %%%% %%%% <[block]{Windows behaviour according to MS} %%%% * The server does NOT enforce same ClientGUID in a session. %%%% * But clients can be expected to do it. %%%% * But it is not explicitly documented like this. %%%% [block]> %%%% %%%% +<2->{ %%%% <[block]{The good news:} %%%% There will be documentation notes: %%%% * Things will not work as expected when clients behave differently. %%%% * It is OK for a server to enforce equality of ClientGUID within session. %%%% [block]> %%%% } %%%% %%%% ==== More digression on ClientGUID : Leases ==== %%%% %%%% <[block]{According to [MS-SMB2]} %%%% * Server Global data Structures: %%%% ** GlobalLeaseTableList indexed by ClientGuid (3.3.1.5) %%%% *** LeaseTable indexed by LeaseKey (3.3.1.11) %%%% * Requesting a lease (3.3.1.4): %%%% ** Object store takes an abstract ClientLeaseId %%%% *** Win7: combination of ClientGuid and LeaseKey %%%% *** Win8+: LeaseKey %%%% * Object store indicates a lease break (3.3.4.7): %%%% ** smb server uses ClientGuid and LeaseKey given by ObjectStore %%%% %%%% ==> !Inconsistent\! -- What to do?! %%%% [block]> %%%% %%%% %%%% ==== Multi-Channel $\in$ Samba : pass by SessionID (plan B) ==== %%%% %%%% <[center] %%%% <<>> %%%% [center]> ==== Multi-Channel $\in$ Samba : Status ==== +<2->{ # messaging rewrite using unix dgm sockets with sendmsg [DONE,4.2] # add fd-passing to messaging [DONE,4.2] # preparations in internal structures [DONE,4.4] # prepare code to cope with multiple channels [DONE,4.4] # implement smbd message to pass a tcp socket [DONE,4.4] # transfer connection in Negotiate (by ClientGUID) [DONE,4.4] # implement session bind [DONE,4.4] # implement channel epoch numbers [DONE,4.4] # implement interface discovery [DONE(linux/conf),4.4] # implement test cases [WIP(isn't it always?... $\smiley$)] # implement fd-passing in socket-wrapper [WIP] # implement lease break replay [TODO] } ==== Multi-Channel $\in$ Samba : Details from @smbXsrv.idl@ ==== %%==== @MSG\_SMBXSRV\_CONNECTION\_PASS@ ==== <[block]{for @MSG\_SMBXSRV\_CONNECTION\_PASS@} <[code] typedef struct { NTTIME initial_connect_time; GUID client_guid; hyper seq_low; DATA_BLOB negotiate_request; } smbXsrv_connection_pass0; [code]> [block]> %%==== Internal Structures (@smbXsrv.idl@) ==== ==== Multi-Channel $\in$ Samba : Details from @smbXsrv.idl@ ==== <[block]{layering before} <[code] smbXsrv_session ->smbXsrv_connection [code]> [block]> <[block]{layering now} <[code] smbXsrv_session ->smbXsrv_client ->smbXsrv_connections [code]> [block]> %%% ==== Multi-Channel $\in$ Samba : Status ==== %%% %%% <[block]{WIP code} %%% * @git://git.samba.org/obnox/samba/samba-obnox.git@ %%% * branch: @master-multi-channel-obnox@ %%% [block]> ==== Multi-Channel $\in$ Samba: TODOs ==== * Replay lease breaks upon channel failure (server $\rightarrow$ client) * teach socket\_wrapper fd-passing ( ==> selftest...) * clustering integration (CTDB) ==== Multi-Channel $\in$ Samba : Clustering/CTDB ==== +<2->{ <[block]{Special considerations} * channels of one session only to one node ! * do not bind connections to CTDB public IPs (can move)! * ==> !add static IPs on public interfaces! \\ % use these for interface discovery [block]> } [frame]> <[sambabg] ==== ====[plain] <[center] \Large Multi-Channel Demo [center]> [frame]> [sambabg]>