You're definitely not alone. Extraordinary insight and creativity, rich, vivid, inspired and out-of-the-box thinking, are typical for "twilight" states, when you're floating between waking state and sleep; not fully awaken to the outside world, but not fully unconscious either. The only problem is replicating and maintaining these states.
It is sometimes possible to remain fully aware in what is essentially a full-on dream state. This is the best of both worlds, combining the rationality and critical thinking of the waking state, with the unbound creativity of the dream. It's like opening up the firehose of new ideas, while remaining in control.
I am fairly certain that at least some of the unusual claims made by people who practice meditation a lot (such as having access to a higher "wisdom", etc.) are based on gaining a measure of control over these states. For this reason, I think it's worth exploring some of the traditional meditation systems, seen as empirical "brain hacking" techniques. I'd love to see more research done in this field; some of Sam Harris' work is relevant.
Thanks for the insight. As a matter of fact, I do meditate :) From my experience (nothing scientific), to replicate these states, what you need is to shut down your rational mind. You see, everyone has in there mind a small voice going "hey, I like this lamp" or "mmm I forgot to do something". This voice is your rational, conscious part of your "self". However, it's not your entire "self". Meditation aims at quieting this "voice" (it's really hard, and I'm not there yet). Once its quiet, you've got "space" for actual inspiration to come in, and once this happens it's amazing, and can happen in many levels. That's why meditation is a mystical experience. Mystical means that it is not possible to communicate via ideas, just by personal experience. If you get good at it you can actually control you emotional states and LISTEN to them, bringing you a lot of wellbeing in many ways. (ok, I'm waaay off topic, sorry).
I like Alan Watts for meditation.[1] I have some mala beads as he suggests, and find they help me a lot. Its easier to time sessions, just count the beads with your fingers and when you reach the end of the mala, you can stop. They also help one to breath/count without consciously focusing on breathing/counting, or even worse, getting stuck in your head trying to focus on not focusing.
Regarding sleep, read something very interesting the other day about Delayed Sleep Phase Disorder (DSPD).[2] Something that might help is a program called f.lux that reduces the blue light in your monitor in the evening, supposedly making it easier it to sleep. I haven't tried it yet but intend to soon.
I'm far from being an expert, but I learned this type of meditation http://en.wikipedia.org/wiki/Transcendental_Meditation
from a guy my mother hired to teach all my family. There are many different meditations and practices, and I've actually never read a book about just meditation but I recommend books by Osho and Deepak Chopra (just google'em) to have insight on Indian practices and philosophy. I would recommend to reach out any yoga institution around town and ask. It's better to learn from someone than from a book, and, as cliché as it sounds, if you are looking for it, the appropriate practice will come to you.
> as cliché as it sounds, if you are looking for it, the appropriate practice will come to you
Yeah. Keep looking for the right career / hobby / spiritual path / etc., and it will likely come to you. Provided you are sincere and are actually looking.
I've a different bias, myself, but to each his own. The various buddhist techniques also seem well suited for the western skeptics, since they typically don't require adherence to any theology at all. You could practice Zen and be a total agnostic or atheist. "Brain hacking" in its purest form. YMMV.
I find Zen difficult, myself. Feels too... um... dry, for lack of a better word. I require something that engages the emotional apparatus, and the physical base, in a more active fashion. I guess this is like preferring Python over Ruby over Perl over etc.etc.etc. Everyone's different.
A few times in my life I've had these between state experiences. I'm not sure what triggered them or why they happened. And I can't recall fully the details of what I saw. However, I can remember the effect it had on my mind.
Once it was morning, but I was not awake. I saw a tablet with writing on it, infinite detail like a fractal going deeper and deeper. But I knew I wasn't awake, my eyes were closed. But the clarity was amazing. My eyes were closed and yet I saw like as if my eyes were open. I told myself this is what having photographic memory must be like.
Another time it was morning again and I thought I was awake. But I saw things floating in the air, like holograms of objects. And somehow I knew, I was still sleeping, but not sleeping. What I saw was a perfect memory recall of my room, a place I had seen thousands of times. But it was all in my mind and my mind could insert impossible objects at will.
Yet another time I heard the most beautiful music. I'm not a musician, and I can't remember anything about the music. I just remember it was amazing, like something bach would create.
These weren't like normal dreams which are forgotten soon after waking. They left a lasting impression, and even though I can't recall the images or the music, somehow I can remember the awe and wonder of the experience.
I sometimes get there by being sleep-deprived. It's pretty hard to get there, though. Sometimes you just fall asleep. And even if you feel creative, you might not have the energy to actually get anything done.
Also, dreams tend to fade rapidly. If you plan to use them to get ideas, it's best to write the ideas down immediately, before the ideas evaporate.
Ah, yes. It's a little bit different. Clarity, calm, a kind of purely-mental energy, an ease to achieve good focus. Sometimes a positive, sunny feeling (unless you're completely exhausted or stressed, in which case it's nasty); other times quite an intense but smooth joy. But at the purely physical level, batteries are running out.
The twilight state has less clarity (it's not obscure, though), and calm doesn't enter the picture. But it's full of this roiling primeval emotional drive to create (in a very wide, general sense). It's where dreams come from, and most inspiring ideas. Some folks find it easier to experience it if they linger in bed in the morning, trying to avoid yanking themselves out of sleep directly into full waking state. It's a fragile thing, handle it gently.
It is sometimes possible to remain fully aware in what is essentially a full-on dream state. This is the best of both worlds, combining the rationality and critical thinking of the waking state, with the unbound creativity of the dream. It's like opening up the firehose of new ideas, while remaining in control.
I am fairly certain that at least some of the unusual claims made by people who practice meditation a lot (such as having access to a higher "wisdom", etc.) are based on gaining a measure of control over these states. For this reason, I think it's worth exploring some of the traditional meditation systems, seen as empirical "brain hacking" techniques. I'd love to see more research done in this field; some of Sam Harris' work is relevant.