fio is a effective tool to test IO-performance of a block device (also file system).
Today, my colleague tell me fio has report “Cache flush bypassed!” which means all IO have bypass the device cache. But I can’t agree because the cache of a RAID card usually could only be change by specific tool (such as MegaCLi), but not a test tool.
By review the code of fio:

static int __file_invalidate_cache(struct thread_data *td, struct fio_file *f,
                                   unsigned long long off,
                                   unsigned long long len)
{
......
        } else if (f->filetype == FIO_TYPE_BD) {
                int retry_count = 0;
                ret = blockdev_invalidate_cache(f);
                while (ret < 0 && errno == EAGAIN && retry_count++ < 25) {
                        /*
                         * Linux multipath devices reject ioctl while
                         * the maps are being updated. That window can
                         * last tens of milliseconds; we'll try up to
                         * a quarter of a second.
                         */
                        usleep(10000);
                        ret = blockdev_invalidate_cache(f);
                }
                if (ret < 0 && errno == EACCES && geteuid()) {
                        if (!root_warn) {
                                log_err("fio: only root may flush block "
                                        "devices. Cache flush bypassed!\n");
                                root_warn = 1;
                        }
                        ret = 0;
                }

and the implementation for blockdev_invalidate_cache() is:

static inline int blockdev_invalidate_cache(struct fio_file *f)
{
        return ioctl(f->fd, BLKFLSBUF);
}

Therefore, the "Cache flush bypassed!" is not mean all IO will bypass the buffer of device, but actually means: "fio can't flush the cache of device, so let's ignore it".
If you want to disable the DRAM cache on the RAID card, the correct way is set cache policy of RAID card to "Write Through":

sudo /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp -WT -Immediate -Lall -aALL