I will explain Infiniband Network in Oracle Exadata in this post.
Read previous Exadata articles to learn What is the Exadata and Why Exadata ?
Infiniband Network of Exadata
Infiniband Network is a high-performance, very low-latency network layer that is active-active in all directions at 40 Gb / sec, which enables communication between the Database Server and Storage Server like following picture.
Full and half Exadata have one main and two secondary Infiniband Switches. In quarter and 1/8 Exadata have only two auxiliary switches.
Infiniband switches provide interconnect between Database Server and Cell Server ( Storage Server ).
Each host and end point has management network port. Cisco 4948 switch as shown above ( in the middle of the picture ) integrates all such management ports inside the rack and provides management network for Exadata.
You can connect to Infiniband switches via ssh, because CentOS linux is installed on Infiniband Switches like following.
[root@devecisw-ibb01 ~]# [root@devecisw-ibb01 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/hda2 471M 265M 183M 60% / tmpfs 250M 24K 249M 1% /dev/shm tmpfs 250M 976K 249M 1% /tmp [root@devecisw-ibb01 ~]#
You can query CentOS release like following.
[root@devecisw-ibb01 ~]# [root@devecisw-ibb01 ~]# cat /etc/*release CentOS release 5.2 (Final) [root@devecisw-ibb01 ~]# [root@devecisw-ibb01 ~]#
You can see all commands that you need for administration of Infiniband switches like following.
[root@devecisw-ibb01 ~]# help all General commands: help - Display this help exit - Exit this shell version - Show switch and service processor version info showfruinfo - Show FRU info for the chassis showpsufru - Show FRU info for the PSUs fwverify - verify switch firmware Switch diagnostics commands: env_test - Do a full environment test showunhealthy - Show unhealthy environment sensors checkboot - Check boot status of Onboard IB devices checkpower - Check status of the PSUs checkvoltages - Check the voltages on the main board getfanspeed - Get the speed of the FANs and check their status showtemps - Show the temperatures showdisk - Show disk's diagnostics information showfree - Show available memory and filesystems space Infiniband diagnostics commands: connector - Show info for a connector dcsport - Map between IB ports and connectors generatetopology - Generate Infiniband topology file getmaster - Show status and history of Subnet Manager getportstatus - Show status for local port getportcounters - Show counters for local port ibdiagnet - IB diagnostic network ibhosts - Show InfiniBand host nodes in topology ibnetstatus - Show status of InfiniBand network ibnodes - Show InfiniBand nodes in topology ibstat - Query basic status of InfiniBand device ibnetdiscover - Discover InfiniBand topology ibswitches - Show InfiniBand switch nodes in topology ibrouters - Show InfiniBand router nodes in topology ibroute - Query InfiniBand switch forwarding tables ibtracert - Trace InfiniBand path ibportstate - Show port state and link speed of an port listlinkup - List all local ports/connectors matchtopology - Match the InfiniBand topology towards a topology file perfquery - Query InfiniBand port counters saquery - query InfiniBand subnet administration attributes showsmlog - Show the InininiBand Subnet manager log showtopology - Show the InfiniBand topology smpquery - query InfiniBand subnet management attributes Subnet Manager administration commands: checktopomax - Check the size of current IB topology enablesm - Enable local Subnet Manager disablesm - Disable local Subnet Manager getmaster - Show status and history of Subnet Manager setsmpriority - Set priority for local Subnet Manager setcontrolledhandover - Set controlled handover for local Subnet Manager setsubnetprefix - Set subnet prefix for local Subnet Manager setsmmkey - Set the Management Key for local Subnet Manager setsmrouting - Set the routing engine for local Subnet Manager showsmlog - Show the InininiBand Subnet manager log smconfigtest - Test local Subnet Manager configuration setmsmlocationmonitor - Enable/disable MSM location monitoring smpartition - Tool for partition configuration for Subnet Managers smsubnetprotection - Configure secret management keys smnodes - Configure which SM nodes that are used for partitioning localmkeypersistence - Manage persistence of local M_Key Other managagement commands: enableswitchport - Enable a switch port disableswitchport - Disable a switch port enablecablelog - Enable syslog of Cable connector hotplug events disablecablelog - Disable syslog of Cable connector hotplug events enablelinklog - Enable syslog of Link Up/Down events disablelinklog - Disable syslog of Link Up/Down events fdconfig - Configure nodes for FabricMIB create_ipoib - Create IPoIB interface delete_ipoib - Delete IPoIB interface createfabric - Configure IP bindings for the Fabric autodisable - Configure port auto disable [root@devecisw-ibb01 ~]#