Infiniband Switches in Oracle Exadata

I will explain Infiniband Network in Oracle Exadata in this post. Let’s review the Exadata Infiniband Switches ( Network ).



Exadata Infiniband Switches ( Network )

Read previous Exadata articles to learn What is the Exadata and Why Exadata ?


Exadata Database Machine -3 What is the Exadata and Why Exadata ?

Compute Node or Database Server in Oracle Exadata


Infiniband Network of Exadata

Infiniband Network is a high-performance, very low-latency network layer that is active-active in all directions at 40 Gb / sec, which enables communication between the Database Server and Storage Server like following picture.

Full and half Exadata have one main and two secondary Infiniband Switches. In quarter and 1/8 Exadata have only two auxiliary switches.




Infiniband switches provide interconnect between Database Server and Cell Server ( Storage Server ).


Exadata Cisco switch

Each host and end point has  management network port. Cisco 4948 switch as shown above ( in the middle of the picture ) integrates all such management ports inside the rack and provides management network for Exadata.


You can connect to Infiniband switches via ssh, because CentOS linux is installed on Infiniband Switches like following.


[root@devecisw-ibb01 ~]# 
[root@devecisw-ibb01 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda2             471M  265M  183M  60% /
tmpfs                 250M   24K  249M   1% /dev/shm
tmpfs                 250M  976K  249M   1% /tmp
[root@devecisw-ibb01 ~]#


You can query CentOS release like following.

[root@devecisw-ibb01 ~]# 
[root@devecisw-ibb01 ~]# cat /etc/*release
CentOS release 5.2 (Final)
[root@devecisw-ibb01 ~]# 
[root@devecisw-ibb01 ~]#


You can see all commands that you need for administration of Infiniband switches like following.

[root@devecisw-ibb01 ~]# help all
General commands:
help - Display this help
exit - Exit this shell
version - Show switch and service processor version info
showfruinfo - Show FRU info for the chassis
showpsufru - Show FRU info for the PSUs
fwverify - verify switch firmware

Switch diagnostics commands:
env_test - Do a full environment test
showunhealthy - Show unhealthy environment sensors
checkboot - Check boot status of Onboard IB devices
checkpower - Check status of the PSUs
checkvoltages - Check the voltages on the main board
getfanspeed - Get the speed of the FANs and check their status
showtemps - Show the temperatures
showdisk - Show disk's diagnostics information
showfree - Show available memory and filesystems space

Infiniband diagnostics commands:
connector - Show info for a connector
dcsport - Map between IB ports and connectors
generatetopology - Generate Infiniband topology file
getmaster - Show status and history of Subnet Manager
getportstatus - Show status for local port
getportcounters - Show counters for local port
ibdiagnet - IB diagnostic network
ibhosts - Show InfiniBand host nodes in topology
ibnetstatus - Show status of InfiniBand network
ibnodes - Show InfiniBand nodes in topology
ibstat - Query basic status of InfiniBand device
ibnetdiscover - Discover InfiniBand topology
ibswitches - Show InfiniBand switch nodes in topology
ibrouters - Show InfiniBand router nodes in topology
ibroute - Query InfiniBand switch forwarding tables
ibtracert - Trace InfiniBand path
ibportstate - Show port state and link speed of an port
listlinkup - List all local ports/connectors
matchtopology - Match the InfiniBand topology towards a topology file
perfquery - Query InfiniBand port counters
saquery - query InfiniBand subnet administration attributes
showsmlog - Show the InininiBand Subnet manager log
showtopology - Show the InfiniBand topology
smpquery - query InfiniBand subnet management attributes

Subnet Manager administration commands:
checktopomax - Check the size of current IB topology
enablesm - Enable local Subnet Manager
disablesm - Disable local Subnet Manager
getmaster - Show status and history of Subnet Manager
setsmpriority - Set priority for local Subnet Manager
setcontrolledhandover - Set controlled handover for local Subnet Manager
setsubnetprefix - Set subnet prefix for local Subnet Manager
setsmmkey - Set the Management Key for local Subnet Manager
setsmrouting - Set the routing engine for local Subnet Manager
showsmlog - Show the InininiBand Subnet manager log
smconfigtest - Test local Subnet Manager configuration
setmsmlocationmonitor - Enable/disable MSM location monitoring
smpartition - Tool for partition configuration for Subnet Managers
smsubnetprotection - Configure secret management keys
smnodes - Configure which SM nodes that are used for partitioning
localmkeypersistence - Manage persistence of local M_Key

Other managagement commands:
enableswitchport - Enable a switch port
disableswitchport - Disable a switch port
enablecablelog - Enable syslog of Cable connector hotplug events
disablecablelog - Disable syslog of Cable connector hotplug events
enablelinklog - Enable syslog of Link Up/Down events
disablelinklog - Disable syslog of Link Up/Down events
fdconfig - Configure nodes for FabricMIB
create_ipoib - Create IPoIB interface
delete_ipoib - Delete IPoIB interface
createfabric - Configure IP bindings for the Fabric
autodisable - Configure port auto disable

[root@devecisw-ibb01 ~]#


Do you want to learn Oracle Database for Beginners, then read the following articles.

Oracle Tutorial | Oracle Database Tutorials for Beginners ( Junior Oracle DBA )





Do you want to learn Exadata detailed, then read the following articles.


About Mehmet Salih Deveci

I am Founder of SysDBASoft IT and IT Tutorial and Certified Expert about Oracle & SQL Server database, Goldengate, Exadata Machine, Oracle Database Appliance administrator with 10+years experience.I have OCA, OCP, OCE RAC Expert Certificates I have worked 100+ Banking, Insurance, Finance, Telco and etc. clients as a Consultant, Insource or Outsource.I have done 200+ Operations in this clients such as Exadata Installation & PoC & Migration & Upgrade, Oracle & SQL Server Database Upgrade, Oracle RAC Installation, SQL Server AlwaysOn Installation, Database Migration, Disaster Recovery, Backup Restore, Performance Tuning, Periodic Healthchecks.I have done 2000+ Table replication with Goldengate or SQL Server Replication tool for DWH Databases in many clients.If you need Oracle DBA, SQL Server DBA, APPS DBA,  Exadata, Goldengate, EBS Consultancy and Training you can send my email adress [email protected].-                                                                                                                                                                                                                                                 -Oracle DBA, SQL Server DBA, APPS DBA,  Exadata, Goldengate, EBS ve linux Danışmanlık ve Eğitim için  [email protected] a mail atabilirsiniz.


  1. After I initially commented I seem to have clicked the -Notify me when new comments are added- checkbox and from now on each time a comment is added I get four emails with the same comment. Perhaps there is a means you can remove me from that service? Many thanks!|

  2. I’d personally also like to mention that most of those that find themselves with no health insurance are usually students, self-employed and people who are jobless. More than half in the uninsured are really under the age of 35. They do not think they are requiring health insurance as they are young in addition to healthy. Their own income is typically spent on real estate, food, and entertainment. Many individuals that do work either entire or part-time are not given insurance by way of their jobs so they proceed without due to rising expense of health insurance in the us. Thanks for the suggestions you talk about through this blog.

Leave a Reply

Your email address will not be published. Required fields are marked *