In everyday life, we experience four states of matter: gas, liquid, solid, and plasma. In extreme conditions, many more other states can be observed, for example, BEC of ultra-cold atoms, superconductivity, and neutron-degenerate matters in a neutron state. In addition, low-dimensional frustrated quantum magnets can provide a promising route to achieve novel states of matter, which have no precedent in a particle system. The paradigmatic instances are spin liquid and spin nematic. Spin liquid is a disordered and entangled state of matter on a quantum level, which breaks neither time-reversal nor spin-rotational symmetry. A spin nematic is a magnetic analog of liquid crystal. It has no magnetic order but still breaks a spin-rotational symmetry in a spin space. In this talk, after reviewing the long-sought efforts of observing such states in a spin system, we will present our on-going experimental results. Specifically, we will provide evidence for spin-orbit coupling tuned spin liquid in the two-dimensional triangular antiferromagnets Ba3Ru1-xIrxTi2O9, and then explore the possibility of multipolar ordering in the FM-AFM frustrated spin chain LiCuSbO4.